MattGPT says:  Which means that if you wanted to go back, and if you're not able to attend class, you'd be able to see the the recordings there. Okay,
MattGPT says: so that's something that I will do every class session. I don't always remember
MattGPT says:  to do it. So
MattGPT says: if you see me starting fast about setting up zoom
MattGPT says: team. Just remind me.
MattGPT says: try and do make sure it's it's going. Okay. So that's that there
MattGPT says:  most of you. How do people ride a bike yet?
MattGPT says: Okay.
MattGPT says: how many people ride? Bikes? Well, have ridden a bike in their life without a helmet?
MattGPT says: Okay, cool. That's good to see.
MattGPT says: All right. Okay, that's cool, because I think you ever tried to do this. But I'm
MattGPT says: twice your age. My! When I was a kid, and I love riding my bike. I still ride my bike. I ride my bike into work into work today. And yes, I was wearing helmet, and I wouldn't dream of getting on my bike without a helmet on these days. Right? I just wouldn't do it. But I remember you know the whole of my childhood
MattGPT says: I wrote a bike. This was in the first sort of Bmx craze. and it never even fought. To wear a to wear a helmet. No, but it's interesting. Now, how like if you go to Google and you can try this if you do a search which is like.
MattGPT says: go to Google images and search for kids, writing wipes about a helmet.
MattGPT says: Try it. I did try a little bit earlier. It's actually hard to find what you're looking for, right? So what we're looking for is.
MattGPT says: I was trying to find a good illustration of a kid having great time on a bike without a helmet.
MattGPT says: kid riding a bike
MattGPT says: without a helmet.
MattGPT says: Okay, so let's try that.
MattGPT says: Okay, well, there is actually a few more than I saw here. But most of what you see actually are pictures of kids with helmets, and if you keep kind of scrolling down, then you see some sort of like.
MattGPT says: I'm not sure if it's going to come up in this one. But there was like pictures of children in ice. The Icu had some sort of accident and stuff like that.
MattGPT says: And it's all about the fact that, like you shouldn't really be writing all up. You know, our cultural sort of
MattGPT says: idea today, at least in in the Us. Is so you should be wearing a helmet when you're on a bike. And kids especially, you know, if you were a parent, then you should be kidding about
MattGPT says: helmet on. You'd probably get some other well, meeting people, probably. Well, meeting parents who would kind of say, why do you let your kid ride without helmet? Okay? That's kind of interesting, right? And how the data is. But if you keep kind of scrolling down what you might find, or if you change the search a little bit.
MattGPT says: or if you were searched for a different country. like, if you were looking at cycling in Amsterdam.
MattGPT says: You'd see all the pictures of people with bikes and none of the way helmets.
MattGPT says: Okay, and so it's kind of interesting. This is what we kind of doing in this course, this is a clust thinking about data. What is data. And we're gonna come back to a lot of things with data is data are, you know, there's that question of singular plural.
MattGPT says: and think, think of, think about a lot of those things. focus
MattGPT says: quite a lot on practical tools by analyzing data. Okay? And so a lot of you may be interested in this class because we're gonna cover a little bit of piping as the main language that we learn. But also some are popular language and some other tools. I'll show you how to use that. So you can get some practical skills which are becoming increasingly important in
MattGPT says: in the workplace, right? Having data skills. It's becoming important
MattGPT says:  But actually.
MattGPT says: we're in the in the context here, this is a communication class, right? So this isn't a programming class.
MattGPT says: honest and engineering for data analysis. But primarily by this is a communication process that we're looking at data science. So we're thinking about where data you know, data are all around us. And we are producing data all the time.
MattGPT says: and particularly thinking about a lot of ways in which we interact and primary primary thing about our interactions as human beings is that we communicate with each other right? And so
MattGPT says:  we're going to. We're going to think a lot about those sort of things right? And so
MattGPT says: there's different ways in which we can see that when we think about doing these kinds of searches and looking how things are represented in the media right?
MattGPT says: Other things that came up when I was doing these search earlier was like an infographic from the Cdc. Which was showing sort of like why you shouldn't ride your bike without a helmet, and so on. And so we've seen
MattGPT says: sort of all those things the other things that we're going to talk about. And I'll talk a little bit about today is rounds in which we encounter data.
MattGPT says: And if we're thinking about being learning some skills for handling data and becoming more comfortable with handling data. And we're going to call that data literacy and send where people read them right? And how that's foundational
MattGPT says: for us. And you know, when we're at school, we learn some
MattGPT says: numeracy or numeric literacy. More and more this term data literacy is being used. And that's sort of like the foundation. And a central goal in this class is to increase our data literacy so that we feel comfortable reading data when we see data represented in different ways, and particularly again from a communication perspective.
MattGPT says: Looking at later in the news where numbers come out.
MattGPT says: etc. Those sorts of things. But also, you know, when we we've got interesting questions.
MattGPT says: how dangerous is it to ride a bike about a helmet? Then there's probably some data out there that we could go and we could try and find. You know, someone's collected some data from er around the around the country, and we can probably find those data.
MattGPT says: And if we're gonna do, if we're gonna ask those kind of questions and wanna shoot them, we just need some basic skills of being comfortable with being able to find those data and understand them structure that they're in. Read them, pull out the bits of information that we need them and then go on and analyze. Okay. So just as we feel comfortable when we say, you know, we've all learned reading and writing like reading allows us to
MattGPT says: process a whole amount of and and
MattGPT says: a whole load of information that's around us. And then we write and we talk to the community.
MattGPT says: by analogy, you want to develop our comfort and our skills of doing the same things with data. Okay, so that's what we mean when we talk about data literacy.
MattGPT says: And some of you may be interested in actually going into careers where you're thinking about be a data scientist or a data analyst. Or I think that within this kind of
MattGPT says: field that I'm interested in, I'm seeing that increasingly, data is becoming really important and being able to handle data is important. And so there, that would be thinking about professional realm in which you're going to be handling data.
MattGPT says: Okay,
MattGPT says: another one is that increasingly, we're going to see data important in the public realm. And we just saw that
MattGPT says: in the past 3 or 4 years. during the pandemic, where
MattGPT says:  on a daily basis. Most of us are exposed to a lot of data about case counts.
MattGPT says:  and vaccination rates state by state location, by location, broken down different ways, and some of that presented visually some of that was presented.
MattGPT says: in articles and increasingly and
MattGPT says: Increasingly, we see this within journalism right. There's what's called data journalism of the developing sort of movement where
MattGPT says: your source for your articles that you're you're making this data, you know, or different types of data.
MattGPT says: And so more and more in the public realm. When there's questions about policy,
MattGPT says: questions up.
MattGPT says: how many accidents are there within the Philadelphia region that involve bicycles or thinking about crime statistics, those sorts of things those data are increasingly available in the public realm.
MattGPT says: And if we're thinking about, you know, issues that we care about and being involved in activism in some ways having
MattGPT says: skills to be able to go and find those data to understand them, to read them, to process them, and then to communicate the findings. There is an important one. And then the final place in which we encamped. Data is is personal. Okay? And so, increasingly.
MattGPT says: you know our whole lives
MattGPT says: without even thinking about it. Where things about us are being recorded all the time right? Carry our phones around? We use
MattGPT says: all sorts of services that are really convenient for us, and allow us to do all sorts of things. And as a trade-off, we agree. Like, yeah, you can collect this kind of data for about me, you know, and I'm willing to trade that kind of
MattGPT says: and
MattGPT says: the privacy for the benefits I get. I use Uber. It's really, really convenient. I use it, you know, finger and app. And you can look at your phone
MattGPT says: on a database. We're using those. And we're giving away all sorts of information about ourselves. And we're saying that that's that's okay. It's the cost of the benefits that we get. So that's the professional round we're going to. We're going to think about where we might encounter data and need be data, literate
MattGPT says: the public realm
MattGPT says: and then the personal realm.
MattGPT says: Okay, so we'll come back and we'll talk about those free areas again and again. And the assignments in this class built around that. Okay?
MattGPT says: And so from my announcement, you saw. And also it's on the syllabus
MattGPT says: and I'm not gonna spend too much time on this. But you see that I requested you think about wearing mark in this part, and my reasons for that. I have to talk individually. People, if they have questions about that are both personal and professional, and so personal is that I live with someone who is
MattGPT says: Immune is immune immunocompromprise. And I'm trying to make sure that I can do the best to protect her from that
MattGPT says: from any any sort of things, and then the professional one is just in my job. My job here is to teach and to.
MattGPT says: you know, give you facilitate this class and help you with that. So I wanna say as fit and help you as we possibly can during this season, which is, gonna be flu season so that we can get through the material so that we can get the most out of it. Okay, so those are the reasons behind that if you're wondering. I appreciate your being respectful of people's decisions on that and you'll see that that's there and link to that. Just as I said.
MattGPT says: the passes will be recorded if you are sick.
MattGPT says: Okay, if you're feeling unwell.
MattGPT says: and you think like alright could be coming down even just with a poll. Right? Like.
MattGPT says: think about that right? Think about each other protecting each other. I don't want to, you know Youtube. It won't penalize you from not being in class.
MattGPT says: Zoom, is there to use that for that? Okay, so
MattGPT says: I'm not gonna spend any more time on that. If you have questions about that, or if you're concerned about anything about that. Just reach out to me. I'll happily talk in more detail about it. Okay.
MattGPT says: great. Okay?
MattGPT says: All right. So
MattGPT says: like, I say this, this week, today and
MattGPT says: first day, we're pretty much getting started and getting set up. Okay,
MattGPT says: just a quick kind of
MattGPT says: sure. And how many people have done any sort of python programming before
MattGPT says: you want? Okay, so there's about 4 or 5 of you. Okay, so most of you, not okay. And that's fine. And that's actually this class of designs, you know, no experiences is no programming. Experience is expected.
MattGPT says: What about Using? The programming language.
MattGPT says: Okay, a few more, maybe sort of like, 7 or 8. Okay? And so those are the 2 main languages that we're gonna cover in in this. Okay? So the
MattGPT says: sort of
MattGPT says: practical in terms of
MattGPT says: exposing you to handling data and working with data using those 2 programming languages. Now, there's other ways in which you don't have to be a programmer to be a data scientist or be a data analyst.
MattGPT says: You know, tools like Excel will do sheets. So
MattGPT says:  more presentation. There's tools like tableau, etc. There's a whole bunch of tools that are out there. Okay? But
MattGPT says:  I do think that having some exposure to doing some basic programming for data, analysis
MattGPT says: is a really useful
MattGPT says: skill to be exposed to, at least. And for some of you may say, Okay, yeah. I had had that class. It wasn't really for me, but at least I understand, and I've been exposed to that. But hopefully, for others, you'll see like.
MattGPT says: well, this is cool. I really enjoy doing this sort of analysis. And learning learning some pipe in and learning some are, will open up all sorts of opportunities for you if you keep. If that's interesting to you, and you keep working on it. Okay,
MattGPT says: I think it's within everyone's grasp. You may think like, Oh, I'm not I'm not a technical person. I don't think I could learn program. I think everyone can to a basic level. Okay, I'm not trying to make you into software engineers or software developers that takes a long time. And that's only for certain types of people.
MattGPT says: But as part of being data literate, which is this skill that we think, you know, we're talking. We're going to be talking about
MattGPT says:  at least, being familiar with how some of these programming languages work, and particularly python and R are the one of the trials dominant within the data realm. Okay,
MattGPT says: and we're gonna we're gonna use this setup. And what I'm showing you here. This is something called Jupiter Hub.
MattGPT says: Okay, which is an online
MattGPT says: platform where each of you, once we get you set up, each of you will have access to to this. And
MattGPT says:  it's like a, it's, it's an online programming environment. Okay? And it makes it really easy for me to give you material to work with.
MattGPT says: And you'll see that there's the course code right there and then it's set up sort of like folders. So what we have over here
MattGPT says: is
MattGPT says:  It's like a flat. File, browsing
MattGPT says: sort of thing. Here's the home folder for me, and you can see that there's a lot of stuff going on in there for you. This is your first class that you've done here. Then you'll just see. Once you get set up, there'll be just one folder for this class.
MattGPT says: and then inside there
MattGPT says:  you'll see. You know, you can navigate around and each class session. Usually I will release another folder, which I'll show you in a minute how you get that. And inside that folder will be some data. Okay, and we'll and then some extra sort of files like imitative.
MattGPT says: And then these These files down here.
MattGPT says: which are called notebooks. And so what a notebook is
MattGPT says: a notebook is a combination that you can think of it as it's basically a web page.
MattGPT says: They can have a combination of code. Okay, so that will be either in our case. That will either be some type of code.
MattGPT says: awesome art code where
MattGPT says: you can put in some code, one line at a time, or a series of lines at a time and again, don't worry. It sounds so confusing. Practice can go through that and then you can actually run or execute those lines of code. So these are, these are the things to do. You run that particular line and then you'll get some output.
MattGPT says: Okay,
MattGPT says: so I'll start a new one right here. So you can see what that looks like.
MattGPT says: Okay. this is what a notebook looks like here on the right.
MattGPT says: And so
MattGPT says: You notice that by default it starts off with as untitled.
MattGPT says: First thing I'll always get you to do when you start a new notebook for yourself a lot of times I'll give you notebooks to work with. But if you do start a new notebook, first thing to do is to always rename it.
MattGPT says: My notebook one very much. That's okay. This is what a notebook looks like. And so you can see by what you get are what are called cells. Okay, all these rows across here.
MattGPT says:  and
MattGPT says: you can type into this little text entry thing here, this is a cell.
MattGPT says: Okay? So by default, you can see that it's a it's expecting some code, and it's expecting some python code.
MattGPT says:  past 12. Okay? You can see, you can put some code in that's actually valid python code. And then when you see that you can either click this button up here, which is to run the selected cell.
MattGPT says: And then you see that you get the output below. Okay? And then it moves on. It automatically adds in a new cell.
MattGPT says: So it's a nice way. It's kind of an interactive way of doing programming. And then we can say, Okay, 7, minus 18. Let's do the same thing. And then you can see that we're getting
MattGPT says: for the output. There's a nice
MattGPT says: easy way to get into learning some basic programming. Okay,
MattGPT says: so that's the first component that you'll see in a notebook.
MattGPT says: But the the more important part of it actually is the non code cells that you can add into here.
MattGPT says: Okay? And so you can see that here. It says, code up here.
MattGPT says: I can click on this cell up here.
MattGPT says: And I can flip it over into what's called a markdown cell. Okay? How many people have ever done any HTML or creating a web page, or anything like that? Anyone
MattGPT says: used to be the kind of thing that was like a middle or high school exercise. And if they do that sort of anymore, okay. But HTML is just a what's called a markup language. It's a way of saying
MattGPT says: I want to lay out. I want to put this content onto a web page, and I want it to be a title. I want it to be in a list of bullets.
MattGPT says: I want it to be in italics, bold, etc. Right on a whole bunch of of other stuff.
MattGPT says: in in a notebook. You can do this kind of thing, but you use. We use a much simpler language, which is called Markdown. Okay? So I put in my first
MattGPT says: notebook here. Okay.
MattGPT says: you can see when I'm in a mockdown. So when I when I execute it. Then you can either hit that little play button, or the thing that you'll quickly learn to do is just press, shift and enter because it makes it much quicker.
MattGPT says: Then it runs it. Now I can go back out and edit. Hit. A markdown has some very simple
MattGPT says: ways of formatting here. So I have
MattGPT says: with a space means make it a first level heading. Okay? So if I then run that, you can see when I do it that I get some some options.
MattGPT says: All the material, like Youtube, will be in notebooks, and some of it will be a lot of information that I'll give you, and those will be in Markdown cells. And that's stuff for you to read through.
MattGPT says: When you see a code cell.
MattGPT says: Then in the version of the notebook that you get, that is something that you can run to see like, okay? So what's the answer. If I do one plus 12, and then you'll get the answer there.
MattGPT says: Okay, so
MattGPT says: those are what notebooks are. We'll come back to that.
MattGPT says:  Do that. Okay? So you can see for today's material.
MattGPT says:  I just want to quickly go over this, this first one, the beginning. Then we'll go into a group exercise. While you're doing that group exercise, I'll make sure that everyone is set up on the server, and then we'll come back again, and we'll try and make sure that everyone can connect.
MattGPT says: and we'll see how far we get through on these other bits of material today.
MattGPT says: If not, we'll pick them up on Thursday. Okay, so
MattGPT says: you'll see over here.
MattGPT says: I can look at this. Okay, so let's take a look at this if you can. Hi, everyone take a look at.
MattGPT says:  this set of parts.
MattGPT says: Could you look at those?
MattGPT says: Add them up and take the square root? There's a data task.
MattGPT says: Okay. No one do it. 3 plus 4 plus 5 plus 6 plus 7 equals. What? And then take the square root fiance.
MattGPT says: Yeah. 5, 5. Great. Yeah. Adds up to 25
MattGPT says: square root of 25. Okay.
MattGPT says: okay? So
MattGPT says: that's the first, you know, I gave you a direct task. There is some data that you're looking at right? It's a set of power.  take another look at their. And what else might you observe? Okay, so I gave you a directed task. But if you were just to come across that, and I was to say, like, What will you tell me about these
MattGPT says: these data that I can you get and yep.
MattGPT says: the parts is one.
MattGPT says: The fiber cause is black.
MattGPT says: Yes, that's great. Actually, most people don't see that straight away. Okay, see that right? When you directly to a task. You probably didn't even notice that you probably switched straight across the numbers. And you were like, Whoa, 25 transfer 25. Yeah. Great anything else
MattGPT says: you can say about them doesn't have to be something that's wrong with them. It's just like some observations on these data, yeah.
MattGPT says: numbers on that the even numbers are red. Yeah. to patent that. Okay, great. And that's a huge part of working with data is being able to look at
MattGPT says: anything else. Yeah. this is just a question. Why is it significant that the 5 of the parts is black?
MattGPT says: Why is it significant?
MattGPT says: You tell me it's I mean? What? Why, it's significant from this, for this example is that you may have missed it until sorry. What's your name.
MattGPT says: Max? Until Max pointed out, maybe I don't know. Did you see it as well? Yeah. So okay, so you thought, that, is that not supposed to be the case? It's well I don't know our hearts usually black.
MattGPT says: Okay, yeah, they're they're not. Okay. So yeah, here's the hub, right? Parts are right?
MattGPT says: So no, no, I'm not sorry. Yeah. But these good observations. Right? So, yeah, the the point here is that, like, you may think like what our eyes are really good. Our visual system is really finely attuned for finding patterns.
MattGPT says: Okay? And that's something that works really well. So when we see this, there's all sorts of patterns there. Right? 3, 4, 5, 6, 7 continuous numbers.
MattGPT says: right? Someone has said, Oh, there's another patent 4 and 5,
MattGPT says: which 4 and 6 read again. Then you can say, Well, there's even numbers. There's another pattern there. right?
MattGPT says: and then it goes black, red, black, red, black.
MattGPT says: So nice kind of sequencing. Happen. Okay? So we're used to seeing automatically these these patterns. And any of you that can psychologists may have heard of like this popular theory about different ways of thinking or processing.
MattGPT says: But Daniel Panama, which is like system one and system 2. Thinking
MattGPT says: system. One thinking is where is like automatic but associative thinking, where we probably are using a lot of. you know it's not. It's it's very quick processing. So we, you know, using our visual system, comes in through our eyes into the back of our brain
MattGPT says: shoot through to the front of brain. And then we kind of we found this patterns. So we pick up a very quick, low, level sort of thing.
MattGPT says: and we don't stop and think about it too much. But we're just like, Oh.
MattGPT says: black, red, black, red, black, sorry black, red, black, red, black. Let me just see that from there
MattGPT says: to pick out this thing here is probably more of a system, 2 kind of thinking which is called deliberative thinking what you do when you add up numbers, maybe when you add one and one together, you are still doing system one. But if I ask you to divide
MattGPT says: 34 by 7 by 15.
MattGPT says: Okay, then you have to think about a bit more and probably go back to some strategies that you have for doing that sort of calculation. Right?  so the whole point here is really just when we come across data, we may immediately think, oh, we know what that says.
MattGPT says: and we know what there is to see there.
MattGPT says: and sometimes we are really directed with that, because we have a task which is like, add up the numbers and give me the square root
MattGPT says: but we also want to develop skills. Intuition, which is like, wait a minute. Is there something else I'm missing here. Is there something else that I can see here?
MattGPT says:  Anything else that people can see there?
MattGPT says: I can shrink this a little bit or
MattGPT says: take a look at the 7
MattGPT says: anyone noticed? Yes.
MattGPT says: yes, right, there are 8 clubs on the 7. So again, if you haven't played out before. There's a suit right? Which is like with clubs, diamonds.
MattGPT says: cetera.
MattGPT says: And then you have that number of things on it right for the number cards here. That's a really hard one to notice. Right? How many other people notice that.
MattGPT says: Okay, you all burn in your development. First time I saw this I was just like, oh, yes, but wanted to just read on to the next bit.
MattGPT says: So that's great already. You're you're having, you know. You've got these good sort of intuitions and skills take home. Then. when we come across data.
MattGPT says: there may be something which immediately comes to mind, or we think that's what the data are telling us, and that may well be in there. But have a second look. Have a further look.
MattGPT says: What I grab! It might look at it this way.
MattGPT says: or if I'm do the equivalent standing on my head to look at these data. Is there something else that I'm missing in there?
MattGPT says:  okay.
MattGPT says: here's some other data.
MattGPT says: 1, 1 0, 3, 2, 0.
MattGPT says: Anyone. What might. What is that?
MattGPT says: Right? 110,320.
MattGPT says: Could be that right?
MattGPT says: Any other ideas? What? That data could be?
MattGPT says: Yeah. November for 2,020. Okay, yes.
MattGPT says: Now,
MattGPT says: yeah. So here we've got month.
MattGPT says: MMVY. YY. Right, you know. That's a fake format.
MattGPT says: How many? Okay you may. You may guess from my accent, although it's changing.
MattGPT says: I'm a bit of a mutt in that. I'm originally from the UK. I'm British but about 3 weeks ago I became an American citizen. So I'm now sort of both
MattGPT says: but of November 2020. But
MattGPT says: before that I would look at that and say, Oh. that's the eleventh of March 20, Crane.
MattGPT says: because in the UK. We go day, day, month, month, month, month, to be a year. And so when I first moved there, and people would ask me on my birthday, I'd say, like
MattGPT says: you know, I give them the day first, and people would put me. I was unborn in May on the tenth of May, right, but that would put me on the fifth of October. Right? Cause I'd keep it the wrong way around. Right?
MattGPT says: So again, the point here is like, we see, we come across data all the time. And these are just data, right? But we need to be able to know a bit more about them to be able to interpret them.
MattGPT says: That could just be a number. But in this case.
MattGPT says: as soon as we see something like that
MattGPT says: goes straight to what you you said. There! It's the date there.
MattGPT says: Which is November the first 2020.
MattGPT says: Anything anything else that someone can tell me about. November the third, 2020
MattGPT says: and November.
MattGPT says: What
MattGPT says: think about something that happens in the thirteenth November?
MattGPT says: Yeah.
MattGPT says: presidential election. That's right. Yeah. First, it's the first Tuesday, I think always right. should know this, because I had to study a civics exam to become a citizen by the first Tuesday of in November every 4 years. And we're on this cycle where it's kind of 2020 was the last
MattGPT says: presidential election.
MattGPT says: Okay? So you can see here that we're moving in a progression from coming across some data like we do all the time. And we're looking at the data sheet. I'm gonna try to understand what it is, we may not have a lot of context for that, and we may have a column title.
MattGPT says: We may have some metadata, some description of these data that says like this is, this is a date. Okay? But even if we have that piece of information, we need to know. Well, where did these data come from
MattGPT says: because is that the eleventh of March? Or is that the third of
MattGPT says: The third of November?
MattGPT says: Okay, so that's kind of interesting to think about right. And if you will be interested in this and Google around on it,
MattGPT says: for us, we probably well, how many people
MattGPT says: had an experience like mine that they learned growing up like a different date ordering system is anyone that can, yeah.
MattGPT says: okay. And so what was yours was that?
MattGPT says: Yeah. So you know, that is something is like same piece of information, right? Or the same piece of information. But it's structured in a different way. And if we don't know the prominence of of these data.
MattGPT says: we could be misinterpreting it. So that's a really important, along with sort of looking and looking again and thinking like, what else am I missing?
MattGPT says: Going from? What? What else? You know? What's the context around these data to pick those up? And actually, if you look around and see.
MattGPT says: the US. Is kind of weird in the in the world. If you look at the
MattGPT says: full month systems, one of the few places that has month, month, day to day.
MattGPT says: Okay.
MattGPT says:  okay. How we doing here?  maybe we'll come back to these once we all get online on that. Okay,
MattGPT says: just this just quickly. Here. Okay, so this question, but not data science, right? We're here focusing on data. Science, certainly 5 years ago. Wasn't it
MattGPT says: the new sexy job? Right? One of the most sexy job that you can have, whatever that means? And certainly this is this idea apart or not. Right? And then I was thinking about this.
MattGPT says: I was like I could be. I could be dating myself. Right. Amazing
MattGPT says: And it's probably not something that people do, because, like it's something I could cover like. And
MattGPT says: and years ago, when that was a thing, and I'm just old and don't move with trends. Very well.
MattGPT says: So we have some really interesting ways
MattGPT says: of looking at these things like Google trends, right? Google trends is a really nice way of getting a quick sense of what might be going on on the basis of the frequency of search over time for a particular item. Okay? So if you Google in the Us
MattGPT says: on Google, I mean, if you go to Google trends, and you look for the term hot or not
MattGPT says: over time. And you say all the way from when you start accepting these data, which is 2,004 to the present.
MattGPT says: Then what Google Trends gives you is a nice plot, a timeline time series of data where we've got time on the X axis here
MattGPT says: and then the Y axis is a little bit weird.
MattGPT says: but what this is showing is that between January the first 2,004. And right now.
MattGPT says:  How many times do people search for the phrase or not? Right? But this isn't what we get on the Y axis. What we get on. The Y-axis is a sort of metric.
MattGPT says: but they don't explain particularly clearly, but it's sort of how popular that search term is how dominant and that search term is so 100 means that's the peak of. During this time, period of when people were searching for hot enough
MattGPT says: or not. So sometimes alright
MattGPT says: 100. But you can see that. Then there's a trend, which is, yeah. It was a trend for a while for a few years, but then, really really take the top to almost the lowest point
MattGPT says: around here.
MattGPT says: But then suddenly, we get these 2 peaks here, or again, 3 peaks. So we get this peak here somewhere in 2,010,
MattGPT says: another one somewhere, maybe in 2,000 13, in the middle of 2,013, then one and 21. Okay. so we'll think about these. And we come across these kind of data all the time.
MattGPT says: Anyone happen to have a sense of like? Why, there may be these kind of stuff, because it seems like, definitely, it's a trend back here in the early 2,000. But then it's like most
MattGPT says: most things it's like, you know, yeah, people don't say that anymore. People don't play that game anymore. People don't think in that way. The language has changed whatever.
MattGPT says: Why would we get some proper people.
MattGPT says: I don't actually know anyone have any ideas.
MattGPT says: Yeah, how can IA general idea? I think that like with pop culture, if it's like mentioned somewhere, ever it becomes relevant again that it's gonna spike. Yeah. And popularity.
MattGPT says: Yeah.
MattGPT says: yeah, I don't know if you're interested, you can try and find and find out when you can go and see like. Was there somebody suddenly who used it? Or was there a new?
MattGPT says: Who knows? Who knows why? That is okay. But we'll come across daylight that and
MattGPT says: want to be able to figure out how to work with that
MattGPT says: and so just quickly. Then, data science, has been distinctly hard.
MattGPT says: And in Sas, we now have a data, science and data analytics minor that you can take data, science and analytics in school of engineering. So even within 10 years become, you know, 3 years ago we had to use the data
MattGPT says: send into the pandemic. But you know we've got. We've got a lot of initiatives now which are around data. Science is certainly within education. And here at Penn, in particular data, science is hot, I don't know has been distinctly hot.
MattGPT says: And as a job for so many years.
MattGPT says: And what I found kind of as a person who is a data scientist and teaches data science and thinks in those areas.
MattGPT says: there's a lot of focus on technical skills. And what do I need to know to be a data scientist. Or you look at Job, add what they want you to know. And certainly you'll see these kind of things like python. Machine learning big data analysis. And now, particularly, you know, finding deep learning
MattGPT says: things like Chat, Qp, generative, AI, etc. So there's a lot of focus on on technical skills.
MattGPT says: But what's interesting? And those may be that, you know, if you go and work at at a Google or a Facebook. Your Meta, where you know, etc. Then, yeah, probably you do need some of these skills at quite a deep level. But most of the people who will go out and do data scientists role
MattGPT says: diva, and some
MattGPT says: enterprise kind of context, or some professional kind of company context where you're dealing with some private data
MattGPT says: or, if you say, went into the public sector. aye.
MattGPT says: those those skills might be important. But a lot of these things, and certainly the the technical sort of trends for stuff are less important, actually.
MattGPT says:  than these skills that we start to think about. Okay. So there's examples with the cards and then thinking about the number, etc.
MattGPT says: Is that really, what is a central skill? Is that a lot of the questions that will be either you'll be focused with when you come across data in these realms. And you remember again, the rounds are professional, public and the personal. The 3 kind of piece that we'll focus on a lot.
MattGPT says: Is that to answer those questions you don't actually need to straight away, throw into a deep learning out room, say, or deep learning model and say, like, I mean what it is. We'll go to chat. Qv. And say, tell me about these data analyzing David, even though you can make it do that.
MattGPT says: But actually, you may be going down the wrong path and do that.
MattGPT says: Okay? So the data questions that need to be answered maybe don't require that much complexity.
MattGPT says: Okay? And what is involved. To get started is being able to collect and organize the data.  and those actually, when you talk to data scientists who most data scientists
MattGPT says: look at these surveys that say that 80% of their time is actually spent trying to get the data, get the data to understand it, to reorganize it, to make sure.
MattGPT says: There are. There aren't missing or errors in it, etc. Those are the main parts and a basic level of description. What is in the data? And then how can we use that data to go from what's in the data, maybe making it useful. I inferential statistics.
MattGPT says: Is is sufficient. Okay, so
MattGPT says: that should be encouraging for you.
MattGPT says: that you don't necessarily need to be anywhere near
MattGPT says: knowing a lot of these really technical things to be a data scientist, to be a data analyst.
MattGPT says: the key thing within this context, it doesn't matter how complicated your analysis may be, what great insights you may have found from whichever techniques that you use. you can't communicate those results in a way that other people can understand and say, Oh.
MattGPT says: if that's useful.
MattGPT says: that's usually right.
MattGPT says: So communication and the insights. And then a non technical audience is II would argue the key skill of being an effective data communicator data, scientist data analyst, whatever you want to call it. They were working with data.
MattGPT says: being able to communicate is the central skills.
MattGPT says: And then from that is this idea that actually, it's not just all about it's intuition.
MattGPT says: along with skills and being able to look at data and get really good at this saying, Oh, yeah, I see that there. And I've seen these kind of data before.
MattGPT says: just as you as you learn to read or you learn to
MattGPT says: you become literate. You know. You don't think about those things anymore. It becomes insure to you.
MattGPT says: And so being able to think about musical theater
MattGPT says: important. And we're going to focus on that
MattGPT says: and call that data electricity.
MattGPT says: Okay, so
MattGPT says: we have.
MattGPT says: I realize, like, I got this wrong. So we've got until about 1140. Right? Is that right? So let's do this. Let's move on to this. Okay, so what we'll do now.
MattGPT says:  in the group exercise. Okay? So just in groups around, you can do like 3 or 4 people. Okay, what I want you to do. Just go around.
MattGPT says: introduce yourself and who you are.
MattGPT says: Give them a piece of data, right? What you had for breakfast.
MattGPT says:  and then
MattGPT says: you can say what your major or what your interests are. And then your interest in data science. Okay, so just quickly go around. And this doesn't need to be really detailed, introduced. I'll get notes of a little bit.
MattGPT says: and then have a quick thing about that in the same way as that, you know. 11. 3 20 number was a little bit like, Oh, first, what is it?
MattGPT says: So you can come up with a piece of data, a number which is something about you. But don't tell people what it is, and then you can see if they can guess what it is. Say 8 years because people would be. Is that how old you are? Right, you know.
MattGPT says: And you could say, like 1, 9, 1, 1 8.
MattGPT says: Is that like the zip code that you're from, or
MattGPT says: those are the tasks for getting to know. You tell it. Okay.
MattGPT says: to do that first
MattGPT says: and then
MattGPT says: once you've done that. Okay, then I'll I'll make sure this is clear. I'll shrink the screen a bit so you can see this. But
MattGPT says: I told you a few things about me already.
MattGPT says: The scary picture of me from a little while back.
MattGPT says: Here are here are some pieces of data about me.
MattGPT says: Okay, and like, I say, I'll bring those so that you can see. Make sure you can see them. There's a whole series of numbers and some labels.
MattGPT says: Okay? And a sequence of numbers, just pieces of data. Okay?
MattGPT says: And then it's a bit like one of those games where it's like maps, you know. Draw the line between the labels and the the data.  these are
MattGPT says: the labels that go with it.
MattGPT says: Okay, so that would be our second task
MattGPT says: to talk with each other to see if you can link these numbers
MattGPT says: with these labels.
MattGPT says: And then
MattGPT says: yeah. And the remaining time, you know. So we're gonna take.
MattGPT says: We're gonna take 15 min to do this, you can talk about
MattGPT says: who had the most interesting breakfast and any questions you have about the class.
MattGPT says: Okay, so that care what to do. before you do that, though, if you haven't gone to the quiz in canvas
MattGPT says: and put in your Github user id do that right now, I'm going to pull those data and get us all set up while you're doing this discussion so that then everyone can get onto Jupiter Hub and get access to this. Okay, if you're not on canvas
MattGPT says: for the class yet, just email, me, your github user. Id.
MattGPT says: okay? So you can make sure that that happens.
MattGPT says: Get yourself into groups, 3 or 4 around you and walk through these things in, okay.
MattGPT says: yeah.
MattGPT says: Yeah.
MattGPT says: Second part.
MattGPT says: okay for us, let's
MattGPT says: great. And
MattGPT says: if you do know, people like get to know some people
MattGPT says: because especially when we get onto sort of like learning the different kind of coding stuff. There's probably gonna be some differences. Some people will find it easier. Some people might have some experience.
MattGPT says: First place, you can go when you're having questions working on the assignments. Is.
MattGPT says: you can work together. You can talk each other. Okay? So I don't have any problem when you're working on assignments. You're working in pairs or in in groups. Okay? Decide. You want to kind of form a study group
MattGPT says: work on the assignment together. That's one of the best ways to learn these kind of skills and to learn
MattGPT says: programming. Okay? The only thing that I don't want, you know, started working together on that. There are some meetings on Thursday, and just to make sure to do the meetings and get some from that there'll be some canvas quizzes which is relatively easy.
MattGPT says: The drug team plus a lot of shore on. Then.
MattGPT says: I asked you to do those one, but everything else in terms of exercises in this class assignments.
MattGPT says: and projects that we do encourage you to to to work together. Okay.
MattGPT says: and final data project where you can work with some public data. You can choose to work on that on your own. But probably you'll get more fun, you know. It'd be more fun, and you'll be able to do more analysis if you pair up at least all get into a group when we come to talk about project stuff.
MattGPT says: You can choose between doing that individually on it group. So
MattGPT says: get to know each other. I'll do that. Okay, let's try a few of these. Okay? So 18,514, what do people think that is?
MattGPT says: I said, Yeah, it's your age.
MattGPT says: it is. I'm 18,000 years old. That's why I'm so wise.
MattGPT says: Yes, that's right. And actually, I am the context for this. This isn't updated to to write today. But yes, if you take that
MattGPT says: leads and and turn that into years. Then you'll get an idea.
MattGPT says: 5, 8. Yeah. So that one's 8.  let's have a look.
MattGPT says: yeah, to this case for me, just to give you a little bit more kind of this pain. Kind of put these together in
MattGPT says: the end of oh, no! The beginning of January this year. Okay.
MattGPT says: the
MattGPT says:  and the class. I was teaching event class. And I hope this. It was an evening class. So I think, overrun as well. That's just made sense of some of these things.
MattGPT says: Okay, 4.
MattGPT says: Okay. We'll come back to that one.
MattGPT says:  29200. 5,200. Sorry done. 95,000,009
MattGPT says: 120,000.
MattGPT says: That's how far I rode on my bike. Obviously in myers. My. any ideas?
MattGPT says: Yeah, your network.
MattGPT says: My network. Yeah, yeah, that's right. No.
MattGPT says: unfortunately, not.
MattGPT says: Thank you for that a little bit.
MattGPT says: It's one of these.
MattGPT says: Oh, I didn't see that
MattGPT says: like it up.
MattGPT says: especially the amount of time I've been working at Penn. Okay, did anyone get that?
MattGPT says: Yep, is it in hours? This is in minutes? Yeah. I've been. I started working here in 2013, which is 10 years ago.
MattGPT says: So if you do the calculation and go from, it's even minutes or seconds. I can't remember that. But, like, you know, do the appropriate thing, and that will come out 10 years. Okay.
MattGPT says: Okay. Then we got some labels. Gilbert Duncan. What were those 2 labels?
MattGPT says: Yep.
MattGPT says: the town you were born in Wisconsin?
MattGPT says: Sorry in Uk, yeah, Duncan.
MattGPT says: that's easy, right? The memories are. It's not the name of my dog? No. And middle man. Yes, it's one of my middle names. Yeah. Oh, yeah, I would guess, for the 4.0 1. Yeah. Well, that I think it's the number of miles you rode the other day on your indoor bike.
MattGPT says: Nope.
MattGPT says: but plus
MattGPT says: it's the number of vars that I said, I ran long time today. When I did this, I was, I went for a four-mile run for lunch. Okay. But yeah, good. What about what about the number of miles on the end of bike?
MattGPT says: Yeah, yes, 15.3.
MattGPT says: Okay. So we're down to. We've got what we got. We've got 4, we've got 21
MattGPT says: again. Remember, this is January. Yes. Yeah. The measure of weather under. Walk about. Yes.
MattGPT says: Any idea what temperature that is.
MattGPT says: Thanks, Elsie.
MattGPT says: No, this is January 21 for that. Right, this is January. So it was treating that. Yep.
MattGPT says: and there's another one given that. Yes.
MattGPT says: so again when someone say, Oh, what's the temperature like? I'm always having to do this when I talk to my parents.
MattGPT says: I can't, you know. I think in Fahrenheit now and
MattGPT says: but they live in Australia, but they also, when they're in the Uk. Oh, it's really hot here today. And then, you know, I'm like, it's 90. And my mom's like, Oh, God, you know. And I'm like, no, I have to do this button.
MattGPT says: and 5.508, whatever it is
MattGPT says: okay. So I think we've got more. apart from the name of my dog. Okay? So the name of my dog, yeah, I obviously
MattGPT says: say, hey? 82, dash 117, dash 100. 200. 1,715. Come here
MattGPT says: right when he's running around the place.
MattGPT says: No, it's not okay. Anyone got any idea what
MattGPT says: my dog's name is. It is those it is those numbers like that.
MattGPT says: Yeah, are those Ascii values? Yes, very good.
MattGPT says: Every major right?
MattGPT says: So a 2 is actually a capital R.
MattGPT says: 100 and 17. Is you lower case, you 102 is an at lower case, F, 117 is another you and 115 is an S. Okay? So his name is Rufus.  And so again, that's when you, when we come across, when we're working with computers, you know, at the base level. They don't know anything about text and stuff like that.
MattGPT says: This is Rufus.
MattGPT says: He is gorgeous. We adopted him about 6 months ago. He is a super mutt.
MattGPT says: I think he is a cow. Mix with a mask mixed with a retriever mixed with something else. Okay, but you'll hear me talk about Rufus a lot because he's our baby.
MattGPT says: Here's another picture of him
MattGPT says: when we've had him a few months, and he's really settled in totally blissing out.
MattGPT says: Okay, so yeah,
MattGPT says: cool. So hopefully, you get to know each other. Any really interesting preference that anyone had in many of the groups.
MattGPT says: And anyone like over that's what they.
MattGPT says: We have breakfast, even. Okay.
MattGPT says:  okay, so we won't. We weren't right on that.
MattGPT says: What I want you to do now is for the remaining 20 min. You can go in your browser to this address.com. They have asd upenn.edu.
MattGPT says: Do it in the browser that you're also logged into Github. And basically you will set up a Github account. Make sure that you're go to github.com, and if you're not signed in sign in there first, all right.  and then go to this address right here.
MattGPT says: Okay? And so then you should.
MattGPT says: What you'll get. What you'll get. Come up
MattGPT says: is
MattGPT says: yeah, it should come up, and it should say something about you for the hub.
MattGPT says: and I will ask you to translate.
MattGPT says: and you just do that once
MattGPT says: take you off the Github, and then it will come back again, and then you should end up seeing something that looks like this.
MattGPT says: So if I want to try that
MattGPT says:  you hit a problem or you don't get that sick of your hand. Okay.
MattGPT says: sometimes this works perfectly lots of times as someone's coming from.
MattGPT says: Okay, so you're gonna go. You're gonna put J. Comjub dot a sc.upenn.edu.
MattGPT says: let's see, actually, do we have another browse on here, and I'll see if I can make it.
MattGPT says: How many people have got to to that so far.
MattGPT says: Okay, looking and good when you do, and you get to that, you should see something that says assignments up here just like I have here.
MattGPT says: Go to that. Simon's list. Okay? And then you'll see there's a pull down here. It should have the course code. I've got multiple ones in there, but you should just see com 2550.
MattGPT says: Select that. Okay. this, this assignment tab is the first place that you'll come. I'd like to begin to say, here, there's some new materials.
MattGPT says: It's called assignments. And these aren't all being, you know, most of the time, actually. And there is a way in which you can submit
MattGPT says: content there. But most of the time I actually just using it to give you content when it is an assignment, and you need to actually submit it. Then I'll tell you about that. Okay, but for most of the time, for the notebook, you just want to use the assignment type of way to get it
MattGPT says: under release assignments. You should see a folder that says 0, on the score of getting underscore started right.
MattGPT says: and then you'll have a button right. Next that says, Fetch hit, fetch.
MattGPT says: and then should turn up over here in the file. Browser.
MattGPT says: Okay, yeah. 2 things. So one it says.
MattGPT says: then, Sig loading, and please wait. So I can't actually like. See what you're seeing.
MattGPT says: And then the second thing is, will this be used primarily where we find our assignments like, should we just like bookmark? This because we'll and this is like more liable in this class than canvas, would you say like, would you say we should come back here a lot.
MattGPT says: Yeah, I mean, every class will be using this right? So all the lecture notes, all of the notebooks will be here. Okay? So the first thing you'll do in class is just go there. Yeah.
MattGPT says: But cameras will use for
MattGPT says: just like announcements, discussions, and that sort of stuff. Okay, for the actual sort of assignments
MattGPT says: into the healthcare. Anyone have some problem getting it come up.
MattGPT says: Yes, that's the same for me. Yeah.
MattGPT says: And
MattGPT says: staying into that dude
MattGPT says: at once.
MattGPT says: That
MattGPT says: okay.
MattGPT says: let's just see.
MattGPT says: Okay.
MattGPT says: what would you? That. It seems like it's the same problem for everyone. I might have to restart the latency if we can fix that
MattGPT says: try and do that with 8.
MattGPT says: Okay, so what I want you to do ignore the assignments. Tab right now. So go to
MattGPT says: you're you'll be in your sort of home folder here.
MattGPT says: let's just let's do a little bit of my.
MattGPT says: okay.
MattGPT says: So if you everyone can, everyone can see this Jupiter hub thing right? Everyone's got that. It's just we're not getting assigned this stuff. I'm gonna have to debug that fix that make sure that's in place.
MattGPT says: But first day, and you know, if I get it working after 5, we've gone up M. And S. Then
MattGPT says: I'll message you say that. But let's just get a little bit familiar with playing around in here because we're gonna do so much in here. So if you go up to the blue plus at the top here, okay, that was that will open up this
MattGPT says: thing called the launcher. Okay.
MattGPT says: the launcher is gives you a bunch of options, but
MattGPT says: different kinds of files within the one that we one
MattGPT says: things that you're gonna focus on most of the time will be notebooks and hyper number
MattGPT says: some other stuff we'll talk about as we go on here for editing files or creating other kinds of files.
MattGPT says: Okay? So if you click on
MattGPT says:  the python 3.8, you should get a new tab that comes up. Okay? So if everyone can do that.
MattGPT says: Okay, so let's just get familiar with
MattGPT says: and over here on the left hand side, you can show. Hide these things.
MattGPT says: We'll talk more about these other things as we go on. Okay. But most of the time it's just you're going to use the file browser here to get around.
MattGPT says: First thing you always want to do is rename your notebook. Okay? So there's different ways in which you can do that. But the easiest way is to right click on the title up here
MattGPT says: and then click rename notebook.
MattGPT says: Leave the extension like, so just change a bit which is highlighted, the dot IPYN, B. So you want to leave. That tells that it's a notebook.
MattGPT says:  okay, so
MattGPT says: my fast
MattGPT says: 9 bucks.
MattGPT says: Okay? So then you should have some of the conditions. Got a title there. and you'll notice over in your file, Browser. Mine's really busy over here, but you can see that that one has just turned up here right? Oh, I'm gonna miss about it. My first, you know book.
MattGPT says: You can also rename things. Do a whole bunch of file operations by right clicking over here.
MattGPT says: Alright. Okay.
MattGPT says: So when you when you start fresh, you'll get a whole series of cells. Okay, so there's a whole bunch of different ways in which you can work with cells. You'll see.
MattGPT says: There's some options in the right hand side here in the cell here. Okay, which moves the cells around, and then you can see this one has a plus, and then a box, and above it, and this one below it.
MattGPT says: So if you click that one, you can see that you can add more cells to your number right? If you click that one, it will add some empty cells. Above that. if by default cells start off as being code cells, okay? And what is a code sell code cell is just
MattGPT says: an entry box where you can put in a piece of python code.
MattGPT says: And it will be treated as code. Okay.
MattGPT says:  so at the basic level, you can use this as a calculator. So you can just play around with putting in some numbers here.
MattGPT says: We've got plus. you know, minus
MattGPT says: division is a forward slash
MattGPT says: and multiplication ever
MattGPT says: asterisk? Okay?
MattGPT says: So you can just build up
MattGPT says: a formula like that like that. Okay, so do something like that. And then if you press
MattGPT says: here like, run the selected cell
MattGPT says: and move on to the next, so we can do that. Notice. You'll pretty quickly want to use the keyboard shortcut right, and that pressing shift, pulling out shift and pressing press pressing. Enter. When you do that, what happens is that piece of piping gets sent off
MattGPT says: to the server. It gets interpreted and run, and then the result comes back. and you'll notice that where this cell started off with a blank like this, this number here tells you the order in which
MattGPT says:  it was wrong.
MattGPT says: Okay? And that's important for a lot of what you do is because in these notebooks you do.
MattGPT says: And when when we're gonna talk about dude scripting or programming.
MattGPT says: it's very much like a recipe. Right? You think like a recipe, it's like you have some basic ingredients. Those are our data.
MattGPT says: And then to make whatever you're making.
MattGPT says: you follow a series of steps which is, first, take these free ingredients.
MattGPT says: environment. Okay, that's a step right? And then take the dry ingredients and mix them into a bowl.
MattGPT says: Then combine the wet ingredients with the dry ingredients, and still thoroughly for 2 min. Those are the kinds of things that we're used to seeing in a recipe. That's exactly what programming is. That's what code is right, we just think about what it is that we're trying to do, particularly when we're working with data. We have some data that we want to work with, and then we say, Well, what do I want to do with it? I want to combine these 2 pieces. Here's a list of numbers which are the heights of people in my data set, I want to know what is the average height.
MattGPT says: So then, you think about how? What's the recipe for calculating the average, you add up all the numbers, and then divide by the number of people
MattGPT says: that will give you an average. Maybe you have an extra piece of information in that data set which is just the height, and whether it's a mail, or, you know, being measured.
MattGPT says: And I want to get the average high age, those categories or some other category right? And so those are really common source of things. So when we're thinking, really, programming is is more about thinking logically in the steps. And what is it that I wanna do? What's the recipe? And coming up with that recipe? And that's about knowing how to do it in one language or another. Okay, so we'll see that sort of over and over again.
MattGPT says: So you can. Just, you know, I just get get familiar with playing around with that. The other thing is that you can switch a code cell
MattGPT says: to a markdown cell whoops not the raw one by going up here and selecting that okay and assume you can see as soon as you switch it over to a markdown cell. Then that that little square bracket, and the colon disappears on the left-hand side. That's because it's not code that will be executed.
MattGPT says: Okay, but you can still put. So then you can put any text into there.
MattGPT says:  this is text, okay? And then when you run that again ship enter.
MattGPT says: then it just gets formatted into that. Okay? And so when it's formatted, if you want to get back and edit it, then you just click on that cell again and you can.
MattGPT says: And I
MattGPT says: down south. Okay?
MattGPT says: So if you haven't kind of come across anything like this, it it can feel a little bit unfamiliar. But the best way to do it is just to play around with it and get used to how it works. Okay,
MattGPT says: Now, if you go over here to the health, there's a whole bunch of health. And the one I wanna just pull attention to is the month down reference. See that one right there. Okay, so if you go down to the Markdown reference.
MattGPT says: so we'll open up
MattGPT says: and show you here.
MattGPT says: This is how you do. So you want to do kind of formatting. You want to turn some text into italics.
MattGPT says: Then you put a single in front of it, and at the end of it, and that will get turned over right? Bold. Then that's 2
MattGPT says: level, one heading level, 2. Heading. How to create links. how to increase increase images and stuff like that. So they're really basic sort of formatting. But the whole idea with Markdown is that you basically just write text
MattGPT says:  but you can easily, then.
MattGPT says: without having to spend a lot of time doing formatting. You can just say, Okay, that's going to be
MattGPT says: and italics.
MattGPT says: Okay? So when you're editing it, you'll see the Markdown codes when you run it. it gets rendered in that way.
MattGPT says: Anything. See that that comes
MattGPT says: so again. I want you just to pray. You won't break anything. Just get me the practicing. It said that if I wanted to have a list of things I can just use a dash. Item one item, 2.
MattGPT says: Oh, great! I get a bulleted list right?
MattGPT says: Let's add another cell above that.
MattGPT says: Turn that into a markdown cell and then a numbered list. You just say one dot. Item, one.
MattGPT says: item 2. Now, if you want a sub-list. you can start another one here. Item, 2.1
MattGPT says: item, 2.2, etc. Okay? And then you see that when you run it it automatically wants it in that one.
MattGPT says: Okay, so it's not super pretty. But you can get a long way very quickly. By just using these basic formatting codes.
MattGPT says: Okay? So if you wanted to create a whole and like a
MattGPT says: a level one heading here, I want to add a cell above this one, so I'll go up there. Oh, this is a code cell. I want to turn it into Markdown. I can either select that here and say Markdown.
MattGPT says: or you'll find that it's actually gets much quicker to click over here where the blue thing is and press M. And that flips it over to Markdown. Okay, so here, set a single hash is level, one heading.
MattGPT says: thank you.
MattGPT says: 22, and so on.
MattGPT says: Now notice if you. If you.
MattGPT says: You need to leave space when you're doing headings right between the code, and then it will try and give you some
MattGPT says:  indication of that in the sense that it doesn't turn blue right? And so then, when I run this
MattGPT says: it does the right thing for the level one and level 2, but the level 3 comes out as plain text. Okay? So
MattGPT says: sometimes you won't get it quite right with a markdown. But you should be able to see that jump back in and fix it.
MattGPT says: Okay, so
MattGPT says: have a play around, have a play around with that.
MattGPT says: The other thing that you can do is that you can upload files into a folder in this space here, and that will be upload. hey? So
MattGPT says: I don't click on any pitches. Yeah.
MattGPT says: no. But let's say you had some pictures or something. Then you can upload a picture and then you can include it in your markup.
MattGPT says: Okay? So between now and and Thursday. If you can make sure that you can log onto this play around in a notebook right at the moment. Yeah, we don't know too much for what we can do. So play around like with that.
MattGPT says: If you know a little bit of python, you can just put any pipeline into here.
MattGPT says: Right? One of the things that we'll learn is often in programming.
MattGPT says: You want to.
MattGPT says:  go over a list of of numbers.
MattGPT says: Okay? And this will be some
MattGPT says: some basic code that we'll learn about. I don't know.
MattGPT says: Print out the square of those numbers.
MattGPT says: Okay.
MattGPT says: so that's the kind of thing very simple steps that we're going to sort of learn is how to do some really basic data processing
MattGPT says: by learning how
MattGPT says: you create these recipes for working with data and then turn them into code.
MattGPT says: Any questions about stuff you're stretching. Okay.
MattGPT says: recipe.
MattGPT says: There's certain subparts of a recipe that you do over and over again. right. like rake an egg and beat it, or you know,
MattGPT says: make it better, or some kind. Okay? So you can. Every time you
MattGPT says: do another cake recipe. You could specify those things again and again and again, hey? But after the after, you know time, you'd be like, Oh, it's just a single I'm going to say it's just a single thing, but it actually has a whole series of subsets to it.
MattGPT says: Okay, and so I kind of, there's something that we'll learn about, which you call functions, which are just little user-defined recipes. Okay, so let's say, I have 2 numbers
MattGPT says: number one, number 2,
MattGPT says: and I'll work. And I want to add those together. Okay, don't worry about this, you know, with the syntax right here. But basically.
MattGPT says: what I'm doing here
MattGPT says: is that I'm defining the the sub recipe for adding 2 numbers together. Okay? And so this is cool defining a function. And you put that somewhere within your notebook. Okay?
MattGPT says: Now, anywhere within this notebook. It now knows that there's a function called. Add that we can use.
MattGPT says: and we can put 2 numbers into it.
MattGPT says: and then we'll get the outset.
MattGPT says: So yeah, that works in there. So
MattGPT says: yeah, notebooks are kind of like, then if you've done any other coding before where you've actually worked in a file, and you have to think about breaking things up. They're a little bit messy from that kind of strict coding practice. But they have the advice which they're really good for exploration. What we're focusing on is working with data and exploring and trying different things out.
MattGPT says:  the the numbers become important
MattGPT says: in terms of the order of executing stuff
MattGPT says: and other questions and stuff.
MattGPT says: Okay, so we finish that today, if hopefully, you know.
MattGPT says:  this sounds kind of interesting to you. I'm excited to have you all in the past. I'll I'll figure out what's going on with that Simon list. Get that working. I'll let you know.
MattGPT says: Thursday. We'll pick this up. Okay, and we'll make sure that everyone can get access and get the assignments
MattGPT says: and we'll talk a little bit more about what it is that we're gonna work on in the class right we'll play around a little bit with getting get hub link in for Jupiter Hub as well. So you can create a a Admin web page. That's what we'll working on, firstly, and then next week we'll start to get into the reading and the actual content.
MattGPT says:  thanks everyone. If there's any questions, just we can always email me.
MattGPT says: get back here as quick as possible. My office hours always through zoom the easiest way for me to do them. I will post some specific hours once. I've time slots, but
MattGPT says: you know it's hard to find ones that work for it once. So and you can always just email me and say, hey? I'm free at like 40'clock on a Wednesday, any time, so we could meet.
MattGPT says: and I'll try my best to do that if you give me some time.
MattGPT says:  you're not yet registered in this class. You're trying to get in. If you want to come down and talk to me, we can figure out what's going on with the wait list.
MattGPT says: Okay?
MattGPT says: I mean, I would be wrong.
MattGPT says: So when oh, that's right, right? Right? So I think we're at 22.
MattGPT says: alright! Let me then
MattGPT says: likely to have an in it worst case like we have a closer 20 thickness.
MattGPT says: probably. Will some people go back? Okay,
MattGPT says: talk with the bachelor. You should get a thing through and you can get on the canvas.
MattGPT says: Okay?
MattGPT says: And alright.
MattGPT says: yes. Are you also in? I am also. Do you want to do by buses, I mean, do you want to?
MattGPT says: I registered or was on one wait list for both, just in case like, I didn't get one or the other. But then the other one sort of came through. Yeah, so like
MattGPT says: might just, I don't know. Still, one or my drop, I don't know. Yeah, I mean, there's some overlap, right? I mean, there is some overlap in this. But this is a 2,000 level twice a week.
MattGPT says: Alright.
MattGPT says: no.
MattGPT says: And that was a good start.
MattGPT says: A
MattGPT says: yeah. I don't know. What do you? I mean you were just in. Oh, okay, well, then, that's not me. I don't know what the next 5
MattGPT says: I know, but like in the media and stuff like that.
MattGPT says: Yeah. him.
MattGPT says: This is Bella just asking me about your thoughts. And should she take it? And I was like, yes, take it.
MattGPT says: Oh, that's.
MattGPT says: The
MattGPT says:  I will always
MattGPT says: try, and I forget to do it. Someone
MattGPT says: can remind me at the beginning and pass
MattGPT says:  just so that
MattGPT says: there's always going to be a class recording of everything that goes on. Okay, and if you are sick.
MattGPT says: or for some reason you can't make fast. Then you can join at same time, or if you can't make it, then you can always come back and get the cross records. Okay? And that way we can
MattGPT says: and get the best of both worlds and stuff
MattGPT says: cool. Alright. So if everyone can open up Browser. And hopefully, I've got everyone. Now.
MattGPT says: Edu. okay, you should
MattGPT says: like that.
MattGPT says: Now, it should be the same browser in which you are. Log into your Github account. If not, what will happen when you go there? Is that the first time you go there it will flip you across the Github.
MattGPT says: and we'll ask you to authenticate. Okay? So if you can do that, once you've got through all of those steps, you should come back to something that looks like this
MattGPT says: as anyone. If you can all try and do that. Is anyone not getting that?
MattGPT says:  that's mute.
MattGPT says: Do you get something like not authorized when you go to that? So that page. That means I still need to add your details.
MattGPT says: Great. Alright. So everyone is looking at something to look like this.
MattGPT says: Yeah, anyone not. Let's do it. And if you're not pick up your hand.
MattGPT says: did you send me? I can't like log into my computer or like, it's like.
MattGPT says: it's like linked to my phone. And my phone's dead. So
MattGPT says: you charge your phone. Yeah, basically.
MattGPT says: Melanie.
MattGPT says: okay. But you did send me your, oh, you did you?
MattGPT says: Alright, someone get you online. I just want to make sure that everyone into the system, because everything we do in this class will be through this through this system. Okay, so that's why we want to make sure that everyone stuff on that
MattGPT says: every class session. Personally, they'll go here. There'll be some material there for us to work with.
MattGPT says: okay, so anyone else not able to get into this. But I like set my github account. Did you email it to me? I put it in the canvas quiz. But I can also email it to you. Can you email it to me again, I'll just double check
MattGPT says: anyone else. Yeah, is there a way to like remove? Because I wasn't your class?
MattGPT says: We'll we'll check on. So if you're able to see this. Okay, there's 2 people who can't yet. Okay, what you gonna do, we'll talk more about what this is. But if you go to assignments
MattGPT says: and an assignment list.
MattGPT says: what I'm hoping that everyone in this class is able to do, and I want to verify right now is that
MattGPT says: you should be able to see in that list come free 31, 80 at 23. Yeah. Do you see that?
MattGPT says: No, I figure it out. I just wanna switch the perfect yeah. Cause I know you're in. You're in the
MattGPT says: 2,100. Yeah. Okay, so
MattGPT says: going whenever we start off this up going to the assignments Tab. And then the assignments list will give you an access, it will say, here you can see release assignments. For now ignore test one. I was just kind of trying to get assistance work, but the one I want you to get is the one getting started. Okay? So if you come over here and I got a lot of other folders in here but yours, probably, if you haven't been in any of these classes before, we just blank here. But as soon as you hit, fetch.
MattGPT says: and that will say fetching. and then it will go down to downloaded assignments. But over here on the left.
MattGPT says: Okay? So what we got here, this is called Jupiter Hub, it's an online programming environment. This part over here on the left is the file browser that works just like explorer on windows, or find our on map, or whatever you use to over here. So folders you've got files and stuff. So you should be able to figure out moving around.
MattGPT says: Go find the folder here
MattGPT says: with the course code on it.
MattGPT says: And then, you see, once you've fetched it in there, then you'll get the material over here.
MattGPT says: Okay, so then we can. So down this tab over there. And if we go inside that folder there. okay, then you can see again, post our borders and some files.
MattGPT says: So usually every every trial session. I'll say, like, Go go fetch the folder from
MattGPT says: the assignments list. Which is this right? And it will be the stuff that we're gonna work on today. Inside that there'll probably be a data folder that will be filed control the data that we're gonna look at. Do some analysis on and then some other sub folders like
MattGPT says: here we have
MattGPT says: an image folder.
MattGPT says: You can see there's a whole bunch of images in there. Just they're organized in that sort of way, and so we'll see some of those and then these ones here on the with the orange square, and that have the IP Y and V, and those are called notebooks.
MattGPT says: Okay, so
MattGPT says: if you want to, if you want to follow. And so then I'll give you this
MattGPT says: at the beginning of the past right? All you could go through these numbers and stuff, but you should grab those, and then you can follow along right? Because what a notebook is right, a notebook is a web page which combines
MattGPT says: text and information which is kind of formatted here. And so the the kind of
MattGPT says: the extent to which these are lectures. These are the like. The lecture notes here, right? And so you can see the stuff to read, and I'll just point out stuff, but it gives you the reference.
MattGPT says: And then there are things which are the screen boxes. And those are called code sets. Okay, so that's what a notebook is made up of, made up of text documentation.
MattGPT says: And so the code cells.
MattGPT says: paying bits of pipeline that you can actually run and see what happens in those. Okay? So
MattGPT says: so as an example. So if you come down to here to this first one. you can tell that it's a code cell.
MattGPT says: right? Because it has these square brackets right here, and then the food on that.
MattGPT says: Okay, that indicates this goes up where these are also cells like this one, right above it. And I can click on that.
MattGPT says: Okay, but that is what's called a markdown cell. And that's just where you put text in.
MattGPT says: Okay, and we'll talk a little bit more about Markdown. Hopefully, you get really comfortable using that, it's a really simple way of formatting text, so that you have headings and
MattGPT says: bullet points and those sorts of things.
MattGPT says: And what we're good at is creating these notebooks as a way to explore data. Okay, so it's a combination of text and description about what it is that you're doing.
MattGPT says: And then so code cells that do the steps
MattGPT says: and that, okay, so those are. You've done a little bit of programming for some of you said that you haven't, and how the
MattGPT says: the analogy that I want you to think about for
MattGPT says: data programming, which is what we're learning is recipes.
MattGPT says: Okay. So who is anyone like the key, Jeff cook here?
MattGPT says: And I'd like to cook?
MattGPT says: Or does anyone have a family recipe that they make? We talked about what you have for lunch.
MattGPT says: Yes, I like to make cheesecake cheesecake. Yeah. Okay. So what goes into cheesecake?
MattGPT says:  I buy the crust already make. I don't do that. Okay,
MattGPT says: I use 2 2 bars of ping cheese.  I believe it's free eggs or something, and
MattGPT says: one tablespoon of gonna extract. I like trying to remember. That's great. You're doing pretty well. Okay. Yeah. Hang out the district grenade.
MattGPT says: Yes, there's sugar. There's like a cup of sugar.
MattGPT says: probably getting the measurements wrong. That's fine.
MattGPT says: That's fantastic. Okay, so come back about how you actually mentioned. But here you can think about this.
MattGPT says: as these are the data. hey? So these are the data.
MattGPT says: the base components for making is cheap.
MattGPT says: Okay, so just quickly, 2 or 3 items. what does that? So I wear a bowl. I put the cream cheese and the sugar together. I mix them until they're fluffy, and
MattGPT says: then I add the eggs, the vanilla extract, and whatever else I'm forgetting  and once I do that I pour it into the cross. Here
MattGPT says: I put
MattGPT says: I put like a pan under it and put some water.
MattGPT says: because that's like a method of baking something. I don't know the name in English. I'm sorry, and then I put it in the oven and wait till it's cooked.
MattGPT says: Well, okay, you just wrote a program
MattGPT says: to work with these data and create a cheesecake. Okay, so yeah, have a think about that. I'll probably offer you one of the assignments. Just fine cause like
MattGPT says: create a recipe, or to to find a recipe, something that you you do and do this kind of thing. Because really, that is what programming it. Right? If we work with data is.
MattGPT says: you have some pieces of data. These are the raw ingredients that go into my cheesecake.
MattGPT says: And then the big, you know the overall level of abstraction sort of like make it, or mix and make, or you know, but you can then break these down into steps right? And these could be broken down into sub steps, too, but sort of first do this, then do this.
MattGPT says: then do this, then do this, then do this. Okay? So that's exactly what we're trying to do when we're thinking about programming, we're gonna start with some data that we're interested in looking at. But we usually need to do is to load it into our notebook so that we work with it. So that would probably usually be our first step, like, load the data in
MattGPT says:  and then we need to understand the structure of the data. Often the data that we come across are going to be in tables.
MattGPT says: have rows. and then columns associated with those which are where
MattGPT says: the measurements for the variables are pieces of data up. And once we have the Sunday, we might want to, you know. Say, we got a data set, which is
MattGPT says: everyone's height in this room. Okay?
MattGPT says:  In fact, I think we might have that data right here.
MattGPT says: Yeah. So if you go into the data folder here.
MattGPT says: Okay? And then you can see, these are examples of data. This is the heights. Everyone in this room. But it's a data set about heights right? Okay? And then you can see that you can open that up in Jupiter. You can see we've got tables where we've got rows.
MattGPT says: and we've got heading to the top. So the first column is the height. some measurement and half inches.
MattGPT says: Biologically, you know, purpose sets label. And
MattGPT says: this is really common biological data. And there's some limitations that obviously do that. But here we've got F's and M's females and males. Okay.
MattGPT says: so those are our data kind of like our ingredients. And then there are some some kinds of analyses that we might want to do on this. The first thing we might want to do is, but you know, once we understand where the data comes from, what's the average height?
MattGPT says: Okay? And so given data that looks like this, how would you calculate the average
MattGPT says: anyone.
MattGPT says: and exactly on average.
MattGPT says: yeah, add them all up and then divide by the number of data points. Right?
MattGPT says: I'll give you an average right? So that would be easy. There's the data. Those are the ingredients
MattGPT says: that's a simple recipe for getting the average time. Add up or the heights, and then, add up all the numbers and divide by the total number.
MattGPT says: Okay, then we can say, okay, well, actually, let's have less. What's the average female height? What's the average male height? How would you modify your recipe to
MattGPT says: do that? Only divided by the number for each like sex will be.
MattGPT says: Yeah. So the best thing to do is split the data into parts. Fs, and M, right? So the roads that have it in them and the rows have M in them. Then repeat them
MattGPT says: step, you know, then tabulated me for each or the averaging, and you've already got a recipe for that. But now you've got half the data.
MattGPT says: Okay? So you know, I sometimes people you know think like, Oh, you know, programming. I'm not good at that or technical stuff
MattGPT says: or I let you know, yeah, pink boxes like.
MattGPT says: arise, and you don't get the result that you're into
MattGPT says:  But the big part and the big picture. And the most important part
MattGPT says: handling data and thinking about doing data analysis from a coding perspective is actually being able to do exactly what you could stop right? A recipe for cheesecake, and then a recipe on a sub recipe for
MattGPT says: analyzing data in that way.
MattGPT says: And so once you figured that out. Then the SIM, that's the hard part. Right? Then. The so yeah, the the easy part. In some ways, the frustrating part in other ways, because you have to translate it into code is to do that, turn into code.
MattGPT says: Okay, so that's what we're looking at here. So in these notebooks, okay, and so when you have here like a code cell
MattGPT says:  you can see this.
MattGPT says: It's got 3 lines of code in it, this and python code.
MattGPT says: hey, Annie? We don't need the one. We don't need to worry right now exactly what those need, but those are put together in one cell, and what you can do is that you can run in a notebook. You can run these cells. Okay? So when you download a a new notebook from me, or you're creating your own.
MattGPT says: Then they'll start off like looking like this. The code cells will have nothing in these parentheses or in these square brackets. But if you click that
MattGPT says: If you click that play button there, or very quickly you'll learn that it's a pain to keep going up here and doing that, and you can see that it's got keyboard shortcut, which is, you just click inside the cell, which is a code cell, and you press shift and you press enter.
MattGPT says: and then you'll get some apple below. Now, here, we just get this orange box. That's actually kind of giving us a bit of a warning, but we tend to it.
MattGPT says: No, those but you can see that that number comes in there which is
MattGPT says: the number of code cells that you've executed or run in the number.
MattGPT says: Okay? And so
MattGPT says: we can keep working through here. Okay? So when I say, you'll follow along with the notebooks, this is what I want you to do. Open them up.
MattGPT says: guide you through. But like where you see code cells, you want to go down to each one and run them and run them in the order they come to.
MattGPT says: Okay. So what we're thinking about here is the simplest way. we can think about stories and data. Okay?
MattGPT says: So yeah, we need some data. And here we're gonna have 2 data points.
MattGPT says: Okay? And so what we've got here, is,
MattGPT says: something called a list. Okay? And
MattGPT says: where we have data, we need some kind of group data together in programming terms that's called a data structure, just a way of organizing data. A list just has a ordered sequence of objects. And here we have 2 labels, A and B,
MattGPT says: and then we're going to call those X here
MattGPT says: less than. And our second list has 2 values, 105 and 302. And we're going to call those Y,
MattGPT says: okay. So we're creating some very simple data here. And again, you just want to run that this isn't going to produce necessarily any out underneath. But you can see with that number free in there, though, as well. Okay.
MattGPT says: now. So then, we might want to ask some questions. About our data, right? Because we're interested in analyzing this.
MattGPT says: We're interested in looking for y values
MattGPT says: here. We've just got 2 values. But it could be a list of 5 values, 10 buddies, 10,000 values, a million values. Okay? And so we might wanna ask a question like, what is the largest value in our data set with a small data set like this was, just look at it and say, Yeah, 102, and the smallest one is 105,
MattGPT says: But again, what programming is just doing is thinking about what it is that we want to know from our data
MattGPT says: and then saying like, how would we do that sometimes we might have think very at a very basic level, which is like.
MattGPT says: start at the beginning of the list, walk through and keep italiate smallest numbers, you see. Then go on. The next one. Is that one small? No? Then go on. The next one. Is that one more? Yes, it is okay. That's the new smallest one. And so on. Okay,
MattGPT says: But often we'll find that we find a happy medium, which is like, Okay, you know, you don't need to tell the computer how to calculate how to find the smallest number in the list. You just use the appropriate function. What is the function?
MattGPT says: Okay? In our recipe. 50 for these steps.
MattGPT says: Okay, and here we've got the, you know, beauty plus sugar. And this is still fluffy. We could break that down to its subsets, which will be, take the beauty state out of the packet, put into the bowl.
MattGPT says: take the sugar pouring, and so it could be more detail. But yeah, we find a happy medium, which is just like, combine sugar and free teeth, and mix and go up
MattGPT says: and so.
MattGPT says: you know, often we'll we'll we'll you know. We'll see that we're looking sort of that way. So environment here, you can see there are 2 functions which are just prepackaged recipes
MattGPT says: for given lists of numbers, they will find the largest number and the smallest number.
MattGPT says: Okay.
MattGPT says: but there's a you know. There's a little bit quite a lot going on in this code here in some ways. Can you see, the main thing here is that we're using the Max function on our list Y, saying, what is the largest value
MattGPT says: and the main function saying list, and it's giving us the small stuff. Okay? Now, if I mess with the data up here.
MattGPT says: and I'll put in 12. Okay? So now we got 3 numbers. Yeah. And then I run this up again. You see, the snap at 12 is this one about about it today? But and that's why
MattGPT says: we wanna, you know. And that's the real event about doing this from code as opposed to, you know, using a tool like Excel, or something like that. It's just that, you know, we're dealing with larger amounts of data and data changes. And we wanna have a a recipe
MattGPT says: series of steps that we can take from one data set and move over to another data set. And then another data set and just adjust it a little bit. And so
MattGPT says: that's what we're learning right here. okay, so
MattGPT says: and we'll, you know. So those are kinds of questions that we might want to ask, what's the largest value? What's the smallest value? We may also have something called the range right? What's the difference between the smaller value and the largest value that tells us something about our distribution of values.
MattGPT says: And again, we could do that by combining these 2 functions, and say, first, calculate the largest number in our list, then calculate the smallest number, and take smallest number away from the largest number, and let's give that a label record, V range
MattGPT says: and then get that okay.
MattGPT says: and so on. And then you'll see that there are some other options that we can do
MattGPT says: Lucas's recipe for calculating how the averages
MattGPT says: add up all the numbers in your list, divide by the number of
MattGPT says: that list. It would add them all up.
MattGPT says: And then there's another function which is called land. Remember, Bunk, when I say function, I just mean this prepackaged code. It's like set in a recipe.
MattGPT says: Len just goes. Count up the number of values in a list.
MattGPT says: so the sum will add the 2 numbers together. and then we'll get the number. In this case 2, and then divide one by the other. And we have meet.
MattGPT says: Okay? And then we have the mean of those 2 values.
MattGPT says: So this is, this is the kind of thing that you know. When I say, when you're learning to code and working with data, starting from this sort of point here. And hopefully, you can see how
MattGPT says: it's pretty straightforward to get that. Okay, trying to use many of examples of this, so that when it comes to
MattGPT says: you then, working on a new data set. It's a little bit like saying, well, I've made it for you. I've made a cheesecake before, but now you're asking me to make a chocolate orange cheesecake.
MattGPT says: So some new ingredients, some new data variants in that. But I'm pretty much
MattGPT says: adapt this recipe. Right?
MattGPT says: Okay? Alright. Calculate the average on heights. Now, you're also gonna calculate the average of scores from the class test. Yeah, I can use the same
MattGPT says: same recipe to do that.
MattGPT says: Okay, so then. but at some point, we're going to want to draw a picture
MattGPT says: and a lot of what goes on in data storytelling. Includes at some point thinking about, are there some ways in which we can visualize these data so that we can look for patterns
MattGPT says: and see if there's an interesting story in that. Okay.
MattGPT says: and python has some really simple ways. Taking some data and turning it into a visualization. Okay, so here we just have our 2 points. Again, remember our list. X is just A and B, there's the labels associated with our 2 numeric piece of data in one.
MattGPT says: And so we can just say, here's another preval recipe plot
MattGPT says: that takes values on the X axis and values on the Y axis. You can change the way that shows them either as a continuous line, or if you put an O in there, then that's going to give us points.
MattGPT says: And then, once we set those out, there's another function, rather, people show which actually doesn't.
MattGPT says: So here we have our same data here. Point A, there's just over a hundred years like the without that reason. Okay.
MattGPT says: and we calculated the average before. So maybe we want to add that mean in there as well. So we can say. here's our data.
MattGPT says: The Me body, right here in the middle is A is okay. So we've got data. We've visualized it. We're starting to think about. Okay, is there a story right now?
MattGPT says: Okay, well, just like when you were talking to each other when you said, Oh, my number is that. And then.
MattGPT says: whatever that is right, at the moment when we we've got these these 2 pieces of data, and we've just got A and B. We don't know anything about them. Okay,
MattGPT says: And so those are just pieces of data. Okay.
MattGPT says: if we come across data like that, it's hard to know how far we go within our analysis unless we have some information associated with them. Okay, so.
MattGPT says: and at the top of the notebook,
MattGPT says: this is illustration which some of you may have seen before. It's quite a common one, which is called like
MattGPT says: the data hierarchy.
MattGPT says:  they pyramid sometimes, and it says that you know you, you have more data to plug
MattGPT says: but to be able to do something like with that, you need to put labels with those pieces of data
MattGPT says: when you have labeled pieces of data, then it comes information.
MattGPT says: Okay?
MattGPT says: So how do you get data somewhere? But once you have the data, you need to look at it and understand it. Put those labels
MattGPT says: on those pieces of data, and that's the interpretation.
MattGPT says: So as you move out the pyramid. You're gaining more insight. But first, you know, just got the raw data. You now have some information about that.
MattGPT says: but then you may have a whole bunch of pieces of information, the heights of people.
MattGPT says: And then we have another piece of information, which is the
MattGPT says: sets of those same people.
MattGPT says: When we combine those 2 like, split them into the 2 groups and calculate the averages. Then what we're doing is that we're looking for an association. Is that some difference between average height of males and females.
MattGPT says: Or whatever the groups are, we might more when we move. Well, that's what we do when we start to look at our data, understand it, and then look for the pants combining 2, 3, 4, 5 variables splitting up our data set.
MattGPT says: What we're trying to do is gain some knowledge out of that. And that's the then
MattGPT says: Truman, okay.
MattGPT says: and that's where a lot of what is in textbooks and a lot of what we do in research stuff is trying to get to this level of
MattGPT says: some new knowledge based on some analysis from the data upwards. And this role of kind of finding associations could be something as simple as just calculating 2 avenues and looking at different.
MattGPT says: It could be calculating correlation, not doing a regression, and so on, might be familiar with that. We'll mention a little bit. and maybe very, very much more complicated sort of data, driven analysis, using.
MattGPT says: I think learning kind of algorithm or something. But at some point we end up here with some new knowledge that we've managed to extract from our data right?
MattGPT says:  But then, in the real world, what we want to do is actually say, Well, what does this give us? How can we apply that knowledge
MattGPT says: in such a way that we can say that now we have some wisdom. Okay, so that's that's that's the data hierarchy.
MattGPT says: You can read through this in more detail. I give you a source that you can look up for that.
MattGPT says: So what we're doing here right now in this next. to excel right here, taking our same data plot, plotting it.
MattGPT says: But now, instead of saying A and B,
MattGPT says: bring in that
MattGPT says:  actually, we'll come. Oh, yeah, sorry.
MattGPT says: Yeah. Now, we put labels on crosses.
MattGPT says: Okay. Now, as soon as we still have 20, as soon as we put labels on our axes, this means something a little bit more. Okay. those 2 values that we didn't know what we were. But we could calculate difference means a lot smallest. Now we've got some information about them that there's actually pollution levels.
MattGPT says: Okay? And then what is point A in point B, unless we had a label, we wouldn't know that could be 2 countries.  you know, that could be
MattGPT says: 2 different companies, terms of the app, or something like that, you know. 2 different cars mix of cars or whatever. But here we've got a label.
MattGPT says: and when we see the label time that kicks in some of our associations with this and the things that we're used to seeing, which is when you have time presented on Xactive.
MattGPT says: In general, at least in the Western sort of English-speaking world energy
MattGPT says: early later. Okay. Now, some of you may speak or read languages that
MattGPT says: go from right to left. Okay, and sometimes you see charts that come from that context where this is early and this is later. Okay. But on the whole, a lot. Most of the charts that you see where time is involved there is something like this earlier.
MattGPT says: Then moving on from that until then, the latest one here.
MattGPT says: Okay, so we can see this.
MattGPT says: So then, given those 2 things. What's the story here in this data?
MattGPT says: What's the story
MattGPT says: yet? Pollution at point A and time was
MattGPT says: lower relative than Point B, like such and such time later, it like, triple. Okay?
MattGPT says: So then, we've got some knowledge from that right? And then the wisdom. Is that right? Well, we better do something about it. Right? Yeah, exactly. Yeah. It's really simple, right? And I'm giving you the simple example. But like, think, yeah, exactly. Exactly. Right.
MattGPT says: So here, pollution is growing has grown worse over whatever this time period is. We don't know where a is. Is it years, is it months? Is it decades whatever? Okay?
MattGPT says:  Now, if we take the same data
MattGPT says: and we have different labels.
MattGPT says: Now, you see. Okay, now, time, we have periods of time, we needed more information. Here. We've got gender as a category. We got A, and we got B, and we've got additional labels. A female fees.
MattGPT says: Now and then. The X axis tells sorry. The Y axis tells us these are reaction times to some sort of psychological test like here, a sign here is sound press, a button. Okay, same data. But what's the story now?
MattGPT says: Yeah, I didn't read that.
MattGPT says: Men have
MattGPT says: whatever was to do.
MattGPT says: And we don't know. We're presuming these are probably milliseconds. I would guess when it's reaction times. But we might need more information there. But yeah.
MattGPT says:  okay. So
MattGPT says: once, we have that sort of thing, we might also then want to go back from our graph and actually calculate something. We're telling a data story.
MattGPT says: right? We can talk about percent change, right? Which is where you've got 2 values.
MattGPT says: You could do this with pollution. But in this case.
MattGPT says: you know, the men's react. You know the average men from the average females. And so we could say 10 and a percentage. Okay? So male subjects were
MattGPT says: 187.6 2% slower than female home. Whatever this cost was.
MattGPT says: That's a way of telling a data story. It's not always
MattGPT says: very complicated. It's not always with a chart. It may just be something like this, a summary sort of number that's made that calculation.
MattGPT says: And you're communicating that.
MattGPT says: Okay? So now let's move on from 2 points to 3 points.
MattGPT says: Okay, just the same, since that's here in Python we define a list.
MattGPT says: We use square brackets, and then each of the items in the list is separated by a comma.
MattGPT says: So we've got 3 labels at C, and then we've got 2 data points, plus, we'll put in one in the middle here by 12.
MattGPT says: Okay? So if we plot those now, we thought. sorry. That looks like this. let's go back to our pollution.
MattGPT says: And then, when we've got 3 day appoints. It's useful to actually join them up. But what's the story here? We're back onto pollution over time.
MattGPT says: Someone else, I mean. Look sorry. Yeah, thank you.
MattGPT says: Pollution has had. It's like a neck
MattGPT says: increases. Yeah, and became 60. Yep. And then we yeah, we go back.
MattGPT says: but
MattGPT says: has went down.
MattGPT says: Yeah. So that's bad news, and that's good news, right? Well, there's good news about you where you're at. You want to tell it in a day's work. Okay?
MattGPT says: But yeah. So the bad news is that pollution has gone up over between time C. And between time A and C. And but the good news is that it got really bad in the middle of that
MattGPT says: 3 A and B. It shot right up. And now it's coming back down again. So we've only got 3 data points. But we can start and look for a trend or a pattern.
MattGPT says: Okay? So when we tell a story is a little bit more complicated, we do want to start an endpoint to the range.
MattGPT says: But first point, we want to say what's going on there. So that's a good news, bad news.
MattGPT says: Let's change our data.
MattGPT says: What about that one?
MattGPT says: Yeah.
MattGPT says: Yeah. Pollution went down a bit. And then it went up a bunch between, yeah, well, yeah, it was looking good, and then went, really bad. Yeah, exactly. But same data points. Then same start and endpoint and change it to slow. Right?
MattGPT says: Okay? So
MattGPT says: those are, those are the sort of things like, give you this example just to show you sort of like getting started thinking about and seeing some pipeline. Okay, that is what it looks like. It's simple. Okay? And it's just like any any sort of learning of a language.
MattGPT says: You gotta start somewhere. I'll give you bits of code
MattGPT says: trying to explain sometimes, and you can always ask your hand and say, what's going on there, or what does that mean? And some people like to learn by trying to understand every bit. But just like when you learn a language and you're trying to start to to learn, you just kind of copy what people tell you to say, and you use it in context. And then all the time, if you use that, use it, you start thinking like, oh, that's not just one chunk of blur. It's actually, you know, a a verb pronoun combination
MattGPT says: again. I can. I understand now a bit more about that. And I can change the verb already. Change the pronoun. And I can understand what's going on.
MattGPT says: Okay, so it's the same sort of thing with with with learning programming. You can't start from the basic principles and try and learn everything
MattGPT says: from the bottom up. I'll give you chunks of stuff.
MattGPT says: You can see what they do, and that's the nice thing about the notebook is that you can run it. You can see what it does, and then you can mess with it. You're not going to break anything, and you can see like, can I add in a full data point.
MattGPT says: and how how would I do that, and so on. Okay, so I wanna encourage you just to be playing around with this
MattGPT says: as we're learning.
MattGPT says: Okay?
MattGPT says: you can give him.
MattGPT says: So we did the group exercise already. We did this backwards from this.
MattGPT says: Let's go.
MattGPT says:  You've seen him, Pop. That's Rufus. my puppy, and that
MattGPT says: I will talk a lot about Rufus and give you a lot of stories about Rupert's years super cute. How many people are dog people?
MattGPT says: Okay, kind of people that cat people.
MattGPT says: Okay, as usual. Well, yeah, okay, we'll actually get some data for this.
MattGPT says: How many people have both?
MattGPT says: Yeah, okay, interesting. Usually, it goes past $60, 30% cap, or something like that, anyway. So Rufus is is is our baby is a mix between chat.
MattGPT says: Retriever, and who knows what else.
MattGPT says: this is, Rufus, a little bit late. That was just when we adopted him about 6 months ago. This is his favorite kind of pose.
MattGPT says: He loves to sort of like just chill out on the sofa and take up the whole sofa. But he needs you also to be sitting there because if you don't
MattGPT says: pay him attention for a little while, I'll pet him, and I'll start to kind of moan at you like it sounds like my wife said. It sounds like a walrus
MattGPT says: and he just wanted him so he's a big cover of bump. He's super super cute. so I will tell you. Some
MattGPT says: tell you lots of stories about Rupert as time goes on.
MattGPT says: Okay, so
MattGPT says: let's keep looking at these notebooks what we'll do so. It's great. I think, that everyone now set up on this
MattGPT says: on Wednesday we'll do a little bit more. So sorry. Let's Tuesday. On Thursday.
MattGPT says: we'll do a little bit more where you can play around with creating your own notebooks, and we'll look at
MattGPT says: creating an about me page, maybe like a recipe recipe page in Markdown flesh
MattGPT says: language and then uploading that to Github. Okay, so that we can get everyone
MattGPT says: on all that. But
MattGPT says: then. So starting next week, we'll start work systematically through some different elements story telling the data some of these skills. And then I'll kind of go through and give you the assignment structure and stuff like that.
MattGPT says: But now I just wanted to give you a few overviews. This will. I want to jump first to this crime incidence and library proximity
MattGPT says: as an example of a
MattGPT says:  a data story.
MattGPT says: So I was working with a colleague mine.
MattGPT says: Jake Pearl, who the data scientist now, who works at Arama, that big company that's across the river by Thirtieth Street station?
MattGPT says: He works as a data scientist in the sport and recreation thing. So basically, I was shared with him that he's doing data analysis about hot dogs. So he's trying to figure out what people buy in session plans, how much beer and how many combinations of hot dogs and stuff like that? But while he was a Phd student here. We we were working, looking at some public data for a while. During pandemic working with an organization for charity
MattGPT says: law firm called an Atmospat Project, and do a lot of amazing work.
MattGPT says: Philadelphia and Pennsylvania
MattGPT says: with incarcerated button. and, as you put, you may or may not know.
MattGPT says: Pennsylvania has a really bad record of incarcerating people.
MattGPT says:  for minor infractions
MattGPT says: often drug related.
MattGPT says: And you know, in the past, and we're talking sort of in 78, even in the nineties and onwards, giving people almost life sentences for that sort of thing. So there's a lot of people who have spent a lot of time incarcerated
MattGPT says:  and most often predominantly. These children.
MattGPT says: And you know they have no, they have no good representation.
MattGPT says: and so this the the involved in
MattGPT says: representing those people and trying to
MattGPT says: the governor has the power to give time to people's therapy, and a lot of times. It's 7 year old people who've been
MattGPT says: populated the whole life.
MattGPT says: The reason we got involved with them was that they were looking for some help with some data analysis when the pandemic came around, because initially, there was no good data about that. But if you think about a correctional facility.
MattGPT says: You know, that's the prime breeding ground for, or any sort of virus airborne disease. So
MattGPT says: and there was no real kind of data being collected about cases there.
MattGPT says: and and a lot of these state and entry state correctional institutions
MattGPT says: go ahead.
MattGPT says: They, you know, there was a whole bunch of older adults. And many of them really unwell. And they, you know, we're still being held. They could have been released. Okay. So there was a push trying to understand these data.
MattGPT says: The weren't really there. Okay? So it started sort of the level of the bottom of the pyramid. But even below that. We haven't got any data that you know. We need to collect some data about what's going on
MattGPT says: in here. and then try and understand these days. See
MattGPT says: what's going on.
MattGPT says: You know, the 2 ways in which
MattGPT says: a disease like Covid could get into a correctional facility would be through new people coming in
MattGPT says: okay, or through the staff who work in the local community.
MattGPT says: And then they're bringing it in. And then we've got a captive audience
MattGPT says: and you know, so trying to understand those sorts of things. And so that's that's kind of side story to what I'm wanna show you here. But basically, we went through a whole series. And this is
MattGPT says: where you know, we had to push and find out where the data was. And at first there was this 24 state
MattGPT says:  correctional facilities.
MattGPT says: And each of them were kind of tabulating in some random spreadsheet cases on a day-by-day basis.
MattGPT says: and then once a week, they kind of email them around to somewhere Central Office.
MattGPT says: And someone would try and piece all of these together into a spreadsheet. And then they would email these around to anyone that might be interested
MattGPT says: in the data. Okay, and then the next day, come on next. You know how it's happened by week first.
MattGPT says: And so someone who, you know it's their job to try and do this. And they they weren't
MattGPT says: expected necessarily to be really skilled data handlers for what they would do with them. But get the new data mess around in the strategy and put new values in. save it with the same file name and then email it around to the same group of people.
MattGPT says: Okay? And they wouldn't. The phone name didn't have the date in it. To say like this was from this date for that date, you know. So there's always missing pieces of information. There was just data that's going around. And so to start with, what we have to do is just get hold of all of these emails and make the assumption that the email came on this day, and then we organize it and do that.
MattGPT says: And then, once we were able to do that, we were able to then move up the hierarchy and put some information labels in there and understand what we took a piece of data men which institutions they were from.
MattGPT says: Whether these were cumulative data, or whether these were new cases and do some transformations on that. And we started to look for associations. In there in in terms of knowledge. Okay, so that's an example of
MattGPT says: a real real world kind of case, example of where  being able to handle data and get hold of data and push to get hold of data that should be available to the public.
MattGPT says: could potentially be really important to make some difference. So again, working with
MattGPT says: the I'm a stock project.
MattGPT says: We had another sort of project that came out of that which was looking at something which is kind of perennial thing amongst politicians in Philadelphia. Right? The end of the year. Sort of or the end of the budget year.
MattGPT says: Sometimes money will become available, and we're talking sort of tens of millions of dollars suddenly become available. And so then politicians will push for like. How can we use that to make positive difference in our community right?
MattGPT says:  and so the variability and peace funding public resources, such as recreation terms, public libraries, are frequently mentioned by anti-variance advocates and right and here's a council member talking about that
MattGPT says: example, the Philadelphia Free Library System. There's real fluctuations in.
MattGPT says: and that
MattGPT says: and the funding. They've been kind of burst the funding which allows them to open 6 or more days a week. And these libraries in this in communities provide a place for people to go to safe which allows people to
MattGPT says: you know, somewhere to go, and so on. And so here's a bigger council member. No coincidence that we talked about it down violence when our neighborhoods were cut off from virtually every resource.
MattGPT says: and so on. Okay, so then it's it's like, that's, you know. I agree that
MattGPT says: things likely to face. But
MattGPT says: because this is such a granular problem. And and it's the the funding is. So you know, someone who wants to try and make a difference. By working with data, we can say, can we add to this rhetoric and to what we believe is true? Right? Yeah, by actually giving them support by analyzing the data.
MattGPT says: And so this provided for us a historical test case.
MattGPT says: It just happened that during 3 months, at the end of 2,019. Most of the branches of the Free Library system in Philadelphia did receive that extra funding in that year. Okay, they went to 6 day a week opening
MattGPT says: but 2 years before.
MattGPT says: In that same time period, the last 3 months of the year, the library did not have extra funding, and the data that we were able to find showed that some of them are only one day a week, some of them just closed sort of permanently. Okay?
MattGPT says: And so from this, you can then formulate a research question
MattGPT says: is, what impact does the presence and acceptability, ie. Being open to say, we public libraries have on the incidence of violent crime, and that could be anything else, too. I would
MattGPT says: pick button, Brian
MattGPT says: but it could be a whole number of things right? That you might want to look at
MattGPT says: and look at the association between these things. So again, moving out that data hierarchy.
MattGPT says: thinking about here in question at the association. Then we need some new knowledge. What new knowledge can we get from the data?
MattGPT says: And so our hypothesis was that switching to 6 day opening will reduce the amount of violent crime
MattGPT says: within a half-mile radius of the library branch. Okay?
MattGPT says:  so this notebook basically just walks you through that and gives you an example of some of the kind of python and data analysis that we're going to learn in this class. And if you want, you can work through and run all these. Okay? And there's a lot of sort of information, at least at the beginning. So, setting up this.
MattGPT says: the data came from open data, Philly side. That will spend a lot of time looking at. It's
MattGPT says: it's good in some ways. It's very patchy in other ways. But there's a whole series of data sets about various things that have been collected out of government offices and services and stuff like that.
MattGPT says: So one of the things that's on data. Billy. Is the prime incident.
MattGPT says: And this is step up late.
MattGPT says: Every crime that responded to, reported and responded to.
MattGPT says: By the Philadelphia Police department gets a line in one of these data sets.
MattGPT says: And so
MattGPT says:  for our particular question, this is where we're sort of like, you know. And these data sets are huge. They've got all sorts of things recorded
MattGPT says: but one of the pieces of data, the labels that they put with each row is the Ucr general Post. Okay? And
MattGPT says: these kind of go within hundreds and then have sub categories doing that like 100 homicide, 101 type of homicide. But second, to be that kind of thing.
MattGPT says:  And so we picked out anything in these 4 ranges, you know, the 100 team. So that's our metric.
MattGPT says: The way that we're going to understand by the crime right?
MattGPT says: So
MattGPT says: to keep working through these and run these.
MattGPT says: we can take it a specific year. So remember, our 2 years were 2,000 2,017 and 2,019.
MattGPT says: Okay, so we can load these data in. Now, these data are geolocated and that each prime has a longitude latitude position in there. And so we'll learn how to do some cool stuff with that data, too.
MattGPT says: But one of the simplest kind of things you can do is just pop these
MattGPT says: use data exploration, and you can see. But it's quite plotting. These are, is this a random sample of a thousand of the incidents? There's a lot more going on there unless hey, cross that with the categories here, and what we're saying is, are there any sort of patterns?
MattGPT says: Not immediately does anybody jump out on that level. Apart from the fact that we do see sort of busters in different places
MattGPT says: visualize the outline of Philadelphia there, and you can see there's some dense clusters in certain neighborhoods and some really sparse points.
MattGPT says: and that would name it.
MattGPT says: So that's our first kind of piece of data that we've got right. But working with this framework, we found that yes, we can get access to crime data. It's much more expansive than we need for our particular research practice. So we need to filter it down, saying, Yeah, we're going to just put the violent crime. So we use the code to do that. We're just interested in these 3 months of these 2 years.
MattGPT says: But we can subset our data in that way. And then where are the libraries? Okay. So there also, fortunately, is a
MattGPT says: well, I figured. Is this true?
MattGPT says: There isn't a data set available? That, or there wasn't a data set available that showed where the libraries were located by Ideo.
MattGPT says: But there was a list. If you go to the Free Library's website, there's the address for each of the branches. And so what we did was write a simple file script that would scrape that, go off that website and get the addresses. And then you can do what's called reverse
MattGPT says: location lookup, which is, give the address, and that will turn that into a longitude and latitude.
MattGPT says: So we now have 50 free libraries in our data set. Okay?
MattGPT says: And we can see if we can take a kind of look at that.
MattGPT says: What we get location and then a free. That's how it's used in this data set be useful for combining
MattGPT says: the name of the library and its position.
MattGPT says: And so that gives us our second piece of data that we can work with.  Alright, and let's take a look at how that looks. and then you get there's the outline. We can also use the phase map with Philadelphia and city
MattGPT says: boundaries.
MattGPT says: you'll find that there are Mass Philadelphia divided into election wards, streets, census units, and so on, or in code. So somehow, you want to think about what's the granularity that I wanna work with data. And then these are the actual position or libraries.
MattGPT says: But I propose that. Okay.
MattGPT says: so what we're trying to do now is that we say, okay, we've got this data right here.
MattGPT says: We've represented it. And remember our crime data 17. We took a sample of it.
MattGPT says: And then we want to say, what's the association right between the location of the library and the number of crimes
MattGPT says: hypothesis it is that presence of a library will have some sort of
MattGPT says: effect on reducing product. If it's totally.
MattGPT says: you know it's a good question to ask it may. And it's obviously very simple. Simplified in terms of there's no direct cause and effect between these things.
MattGPT says: Okay, so then, what we want to do is say, well, we don't just want to have the point there.
MattGPT says: This is a random book. Say half a mile. We do, you know, change that we drew. We drew a half mile
MattGPT says: buffer around each library, and then you can see what we're trying to do. And again, like, I'm showing you the python code here.
MattGPT says: Use these chunks, and you'll be able to, I hope, by the time we get to our data project to do something similar
MattGPT says: yourself. But really it's it's the high level that I want you to get here, which is like
MattGPT says: breaking down problems, is like the influence of the library. Well, we need to operationalize that in some way. What's the sphere of influence? Well, let's just draw a half a month, and then, once you've got these masks crime incidents with these data, and count up the number of violent crimes within each of those. So
MattGPT says: okay? And so then remember our time windows. October December 2017, Covid, December 2019.  there's a whole series
MattGPT says: code here. But basically what we're doing here. And I should have documented this better here. But basically. this is the recipe. All students said. load the data.
MattGPT says: create the buffers around that. Then the 2019
MattGPT says: combine, load the data and just take violent crimes.  and then
MattGPT says: subset the data so that you just have our approval to December.
MattGPT says:  and then
MattGPT says: here this is webcam which is combining or joining the 2 data sets. So the library data set with the radiator, with the circles in them.
MattGPT says: and then 23 and 2019 and only keep the crimes that occur within those.
MattGPT says: Okay? And so then we can look at that within a table. Here's the 2017
MattGPT says: within the sphere of influence of the Andorra library that were to buy a crimes in those 3 months in 2,017.
MattGPT says: And so. okay, so we can, we can just count up those
MattGPT says: and then you can see. Well, the metric is based on our logic, but what we're hoping is, if you take away
MattGPT says: the number of crimes within each library in 2019 by the 2017. That would be a decrease
MattGPT says: if the effect of opening a library is such that it reduces the likelihood. Right? Okay? So that's our recipe. Again.
MattGPT says: code looks a little bit woo. But it's basically just doing that. What we're trying to do is take one away from the other.
MattGPT says: And then we want to express that as a change, the percentage change. Okay, so it's a positive number. It means an increase in prime
MattGPT says: between 2017 and 2019, a negative number means a decrease.
MattGPT says: Okay? So we could look at that as a table. We could order, the table rank the table.
MattGPT says: We could also think of various visualizations.
MattGPT says: This is a verbal Baja middle point here at 0. Right?
MattGPT says: So no change. And then to the right would be an increase, and to the left would be
MattGPT says: not the prettiest graph in the world. And again, you have to know, like, there's a lot of things on the y-axis
MattGPT says: and anyone tell me why or what's disastrous about these particular colors?
MattGPT says: Yeah.
MattGPT says: Fred and Blue me, I mean, I see that.
MattGPT says: Think about again as we look at data and look at again, think about seems like, Oh, Democratic something Republicans. And that's perfectly, you know, that's become established. If those sorts of associations are there for people, then we want to understand them and avoid them when we do our visualization. So yeah.
MattGPT says: I'll get that right?
MattGPT says: Okay? So coming back, yeah. But we, we could look at this, and we could count up like, so where have there been increases. Where have there been decreases? We could also just do that programmatically.
MattGPT says: okay, and so we're interested in where the change is sent smaller than 0. And then counting up, how many of those are? Okay? So we have picture 3.
MattGPT says:  yeah. And so yeah, it doesn't seem like it's strong argument. That's almost 50 50, right? A few few instances more where there is a change to the negative side of things.
MattGPT says: So there's no marked pattern of reduction there and then the final step.
MattGPT says: It's just in trying to communicate these data. We could start to move towards something that like we may
MattGPT says: wanna work through a little bit and and explain in a data story and the
MattGPT says:  change of mind. Incidents of 6 day library remains
MattGPT says: few periods compared, and then it would go back 18
MattGPT says: not, or he change here. So when we when things go red and that production when they get through, then that's
MattGPT says: I guess there's no real fear about patent
MattGPT says:  And I, you know, thinking about it as a pretty simple. basic idea, isn't it, that just by having the library, but you did it.
MattGPT says: But you see, like the steps that we walk through in doing that to come to at least test out that.
MattGPT says: Okay, so this is a kind of kind of thing that we're going to cover in this class.
MattGPT says: how we can access these data, formulate these questions.
MattGPT says: what's missing from this particular? This is the data analysis, right? Step by step, working through this.
MattGPT says: And then we'd need to turn this into a data story.
MattGPT says: So it would go with data analysis. How am I going to tell you that story? Now? I try to tell you that story
MattGPT says: little bit long wind and way, and I can kind of focus on sort of the steps. But if I was gonna try and communicate that some and say, Hey, what have you been working on
MattGPT says:  then, or if I was going to try and go to that council member and say, Hey, you know I agree with you. This is really important. When we want to push for this.
MattGPT says: we tried to look at the data. And this is what we found.
MattGPT says:  is the way to present that.
MattGPT says: Wow! Aye.
MattGPT says: cool any questions.
MattGPT says: we're just about at time today. But so
MattGPT says: if you wanna.
MattGPT says: I would encourage you in your own time have a look over the second one here. Okay, this is another notebook, which again goes, looks at open data.
MattGPT says:  and you can work through it cell by cell, right? Read through. Get a sense of what's going on
MattGPT says: here. We're working with crash data. And upon the castle
MattGPT says: between 2,007 2,017. And it shows you how the steps work for loading the data, understanding the data.
MattGPT says: subseting the data and then asking some questions from it. Okay? And it does some plotting and stuff like that as well.
MattGPT says: So again, I want, you know, it's it's kind of in the same way as sort of
MattGPT says: the way to learn French is the town on the media and into a French soap opera
MattGPT says: good way to stop. Get comfortable with data programming
MattGPT says: mixing these 2 things together.
MattGPT says: Great. Alright, thanks. Everyone. Thursday will
MattGPT says: talk more about what's going on in the class, and we'll show you how to work with the assignments.
MattGPT says: we'll fill. I'll fill out this schedule so that we can. Okay, thanks everyone.
MattGPT says: These. We only don't
MattGPT says: about empty
MattGPT says: if you if you weren't able to get onto the survey, just come down and we'll get you set up.
MattGPT says: Over 9, let's say
MattGPT says: you should see. The feedback for assignment. 2. Let me know if you don't see it, because the feedback system doesn't
MattGPT says: always work directly with the fetch feedback, but if you don't see it, then I can. You let me know that I can copy that forward into your assignment to folder
MattGPT says: working through assignment free.
MattGPT says: Try and get that done for Thursday so that we can look over that. Then
MattGPT says: I don't know. because if you should've had the feedback from the quiz as well as we.
MattGPT says: everything everyone's doing. it's doing really. Well, hopefully, you're finding the readings useful, this sort of structure for
MattGPT says: covering sort of the basic concept of data literacy. And now we're moving into sort of state of visualization.
MattGPT says: And some of those concepts that we covered. We got in Chapter 2 of learning to see data that we talked about last week in terms of the visual system.
MattGPT says: and how we respond to different types of markings or digital stimuli
MattGPT says: length. And then things like area and angles and volume. we'll we'll practice more, and we'll talk more about those on Thursday.
MattGPT says: Well, on the assignment this week you saw
MattGPT says: we were just you were doing some plots using Map Plotlib, which has been the base library that we've been using so far. But that final question where you had the 2 maps that's using a library called Plot me, which is
MattGPT says: really nice library that works in both our and python
MattGPT says: to do interactive graphics. So you see, when you plot some data there it automatically gives you like a mouse, over which will then show you a tool tip which shows you values and stuff like that. So sort of moving towards some of the stuff that we we saw with dashboards
MattGPT says: really easy. Once you've got your data on you, you understand your data, the user, a library like that, and the the syntax that we didn't think for creating power chart.
MattGPT says: It's kind of a line pause
MattGPT says: moves over pretty easily, so we'll look at that on
MattGPT says: thursday a little bit. And the other thing I wanna do on Thursday is
MattGPT says: book about the annotated bibliography for your short essay on a daily public
MattGPT says: be due next Wednesday.
MattGPT says: 8 pen sources. Okay?
MattGPT says: which is giving you a sort of preparation. So you want to start thinking about your topic for your short paper, and I'll
MattGPT says: certainly on Thursday, but maybe I'll try and post them on canvas before show you the titles and a description, you know, of the papers that people have done before. So you can get our idea of the kinds of topics that people have covered. But we talked about this a little bit before
MattGPT says: find something that really interests you, which is
MattGPT says: somewhat. You know the intersection, maybe, of ethics and data.
MattGPT says: if you're interested in a specific area like marketing or so. and sports communication or something like that, you can. You can talk about
MattGPT says: something in that realm
MattGPT says: people have written about.
MattGPT says: you know, the role of data could be playing in a particular industry that they're interested in, or that is playing
MattGPT says: interaction of ethics and
MattGPT says: private and personal data. Those sorts of things make really good papers. Okay, so we'll talk more about that on Thursday.  and then. So yeah, we're doing. I think
MattGPT says: it's amazing that we're in week 6 already. So today, we're just gonna kind of go
MattGPT says: keep working through pandas and talking about working with tabular data
MattGPT says: and the private the public data domain that Jones talked about in Chapter 2,
MattGPT says: okay, which is
MattGPT says: where we're going to be working for our data project at the end. As you know, as we move through things here.
MattGPT says: so yeah, firstly, we'll cover those things. this week, as usual. There's a quiz on Friday, and then there'll be a time on it.
MattGPT says: for on Sunday building on one to sign up free. So working with pandas.
MattGPT says: working with some open data. So practice that today and on Thursday as well.
MattGPT says: And then next week is
MattGPT says: you know, next week is really supposed to be full break, but you know full break. It's narrow.
MattGPT says: We connect.
MattGPT says: but I guess because of Thanksgiving. But what what we're gonna do. So you'll see there's no class. On Thursday. The Annabelle bibliography is due on Wednesday before the beginning of full break right here.
MattGPT says: But next Tuesday's class is really you know, we're
MattGPT says: kind of in the middle
MattGPT says: stuff. So I want to do a review of what we've covered so far. And so
MattGPT says: I think overall, everyone is doing really great. I'm really impressed with how people have picked up the coding and the data analysis concepts.
MattGPT says: but I want to give a chance sort of for a bit of a pause. So next Tuesday's class will actually be on zoom. Okay?
MattGPT says: and what I plan to do there is sort of go over. Give a review of of what we've done, but also kind of treat us as a chance for you to come along and ask questions.
MattGPT says: If you are here, and you want to come to the classroom and do that and be but just join through the the zoom link. Okay, so that's next Tuesday's class.
MattGPT says: And aside from the we'll get started on, and it shouldn't take you that long. Once you pick your topic.
MattGPT says: Okay? Then
MattGPT says: there is no quiz and no assignment next week. Okay? And then we come back after full break. And
MattGPT says: we'll continue looking, working with public data which will start today, and that will give you sort of hopefully. There, you'll start to generate some ideas for your data project.
MattGPT says: and I can also share with you some projects from previous classes. What kinds of things people have done. We'll continue working with pandas
MattGPT says: and then there'll be a quiz that we continue up with the meetings, and then the end of that week.
MattGPT says: Yeah, thanks.
MattGPT says: And then from week 9. We
MattGPT says: continue kind of working with data. In tabular form. But thinking about time series data, okay? And this is going to be relevant for your quantified self software. You've been collecting data on yourself every day hopefully, do you need to keep doing that?
MattGPT says: And so we'll start off by seeing how you can do that kind of thing in pandas.
MattGPT says: But that would be a good place to just like to introduce a little bit of our
MattGPT says: we're not gonna spend as much time working with our as we have with pandas. Sorry with Piper and pandas. But the concepts that we've learned.
MattGPT says: We'll transfer over really easily. Okay. So we'll learn some plotting, using Gg plot, which is a popular library for doing plotting in our
MattGPT says: and something called tidyverse, which is a data flow data analysis, sort of library that works in in our very similar concepts. But what we've got in pandas. But you can see how to do those in our
MattGPT says: and then for your quantified self assignment. You'll actually do that using the our stuff that we learn. Okay, there's a way to practice it.
MattGPT says: So just a sort of show you the lay of the land where we're going.
MattGPT says:  any questions on anything in terms of the assignments, quizzes.
MattGPT says: That's 58, yeah.
MattGPT says: So just to double check the next Wednesday, that assignment is just the and maybe not including the actual
MattGPT says: usage. That's right. So it's kind of getting ready for that, do you? Wanna you wanna pick a topic right for your paper. But that's you.
MattGPT says: Monday. What would that be?
MattGPT says: So the end of week? 8. Right? So the Anthony big Orphe is Jeff.
MattGPT says: 8 to 10 sources. Okay, so there you pick a topic which is like Chat, Gpd. And education right? What are the implications for that? And you make it high school education or middle school education? So you know, as usual.
MattGPT says: just a short essay. So don't be too general. If I mean
MattGPT says: interesting to you, and we'll just target it a little bit.
MattGPT says: What I want to see in the annotated biography is and sources, and at least 3 of those finding academic from a journal article or something like that. Okay.
MattGPT says: and then if you haven't done an anti video, or you just gonna search for those articles. But you know, outside of those free academic ones, it could be that, you know you pick somewhere there's a blog post, or there's a news news article
MattGPT says: or even just a popular book where you find something that's relevant to it. Right? So then, for now a bibliography, you just need to get the photographic information for that source and then write a paragraph that summarizes
MattGPT says: key arguments there, and particularly, you know, with relevance for what you're doing there. So just be a paragraph for each of those.
MattGPT says:  and we'll talk about how to you know how to do that, and how to lay it out. And then you'll just submit it.
MattGPT says: probably I'll put a Jupiter Hub assignment! I'll put a Jupiter hub assignment for you, and then you're just up to to work in, and I'll show you how to work in that
MattGPT says: any other questions?
MattGPT says: great. Alright. So let's take a look at
MattGPT says:  the folder for today. So there's a whole bunch of notebooks in there. We're not going to get through all of those
MattGPT says: first one I just wanted to.
MattGPT says: Put for everyone to see. These are most of the links, you know the quiz this week.
MattGPT says: You define the tax board.
MattGPT says: play with it a little bit, and then to describe, you know, what kind of people it was aimed at, what kind of questions could be asked of it. So here's just a
MattGPT says: the links. The that different people found us, and really interesting ones in there that I hadn't seen before.
MattGPT says: Encourage you to take a look at that and see how those different dashboards
MattGPT says: work, and you know, think critically how well that they work or don't work where they're confusing.
MattGPT says:  the first notebook here. Number one.
MattGPT says:  yes.
MattGPT says: to give you sort of a.
MattGPT says: not a cheat cheat. So much is just the review of the basic stuff that we've covered in in pandas so far. Okay?
MattGPT says: And in assignment 2.
MattGPT says: You were working with the candy data set right, which was, you know, the top ranked candy in different states.  but the way that we worked with it was just in free lists like this, right? So this is really showing how we can transition from art
MattGPT says: basic data structures
MattGPT says: ordered sequence of objects, these free lists that we had. And we saw. Yeah, we can work with that. We can zip them together. We can loop over them.
MattGPT says: We can use indexing to get into that.
MattGPT says: But as we get in, you know, really
MattGPT says: that is, a table should be, you know, set up as a table, and it would be much easier to work with it in table form.
MattGPT says:  so what this notebook does is just give you a sort of a review of
MattGPT says: the basics that we've covered so far. So if we import pandas.
MattGPT says: then we import our data right? We've got our free lists for our columns. And then you created this dictionary to practice working with dictionaries, which is that each of the candies the key was the name with the candy name, and the value of either through a pulse where a head chopper in it. Chocolate-based candy.
MattGPT says:  okay. And so,
MattGPT says: usually, when we're working with data frames, we have a Csv file or excel file, or some format where we can learn, including standards.
MattGPT says: But if you.
MattGPT says: if you create a dictionary like we've seen before, where you've got lists as the values. And then the key is the column name
MattGPT says: I did.
MattGPT says: How we work with a dictionary like that when we were looking at the rainfall in Philadelphia over time, you can actually take that dictionary and using the data frame
MattGPT says:  objects, pandas data frame object, you can turn a dictionary into a data frame. Okay? So that's just what that's doing. Right there.
MattGPT says: we're on.
MattGPT says: Hello.
MattGPT says: There we go.
MattGPT says: Okay. So you can see that we go from a
MattGPT says: dictionary here, where he is the following name.
MattGPT says: and then the value is the list that we find up above, and it turns those into a database. So now we have this index. This is our row index. Remember and we have all 3 columns there. We can do call indexing by the same
MattGPT says: for that thing.
MattGPT says: Column brackets, state, and so on. And so this is a much better way for us to work with data
MattGPT says: in a tabular form like this.
MattGPT says: Okay? But usually we want to create a data like that usually will have a data source
MattGPT says: data file. And we've got the candy here. and remember we can open it either by default. It will open up in the table view
MattGPT says: if you right, click on that and say, open with the editor instead. And you see, this is what a Csv. 5
MattGPT says: for, but it's the simplest hectic file for encoding people.
MattGPT says: The first line of the file
MattGPT says: has comma separated values where each value is the column name
MattGPT says: and then line 2 through
MattGPT says: the end of the file. It's just each of the rows. So did not wait.
MattGPT says: Okay? So usually, we'll come across later in this form.
MattGPT says: And so we load it in with our
MattGPT says: Psv
MattGPT says: data file folder and load the file. Okay, so person will always do is either create the data in a data frame.
MattGPT says: but usually load it in this way and then get the dimensions of the data. Okay? So,
MattGPT says: how big is the data? How many roads does it have? How many phones does it have
MattGPT says: we use data frame of shape. We can pull those out into the 2 objects.
MattGPT says: Good way to do that.
MattGPT says: let's yeah
MattGPT says: one, or we can do something pretty neat like this, which is
MattGPT says: load the data columns.
MattGPT says: And then
MattGPT says: you have a sense. So we got 51 roads. So 51 states representing our data and free columns.
MattGPT says: When we got bigger data, then we want to start to inspect it. They will want to.
MattGPT says: What does each row represent? What's on each column.
MattGPT says: so head by default will give us the top 5 rows to the first 5 rows in the order that's been in the file.
MattGPT says: tailed at the bottom, and and spent quite a number of both heads. else at the bottom 7 rows would be like that.
MattGPT says:  And then
MattGPT says: a good thing we've got a really large data frame is just take a sample every time we run. sample will get another random sampler problems. Okay? So it gives us a good way. This data is is super consistent
MattGPT says: things. And it was members
MattGPT says: working with much bigger data sets, will finding consistencies just by taking random samples out.
MattGPT says: okay, so and then remember, we can get another way to when we have a lot of columns, and they don't all show up when we look at the head tail or the sample. We can actually list all of the columns by just saying columns.
MattGPT says: okay.
MattGPT says: Now, remember our dictionary that we created in assignment to where the key was the name of the candy, and it was the value, either true or false, so we could use that as a lookup
MattGPT says: is chocolate, and then the candy name, which is the key, and then we'll go back to value
MattGPT says: right? And then the same snickers.  But actually, what we want is
MattGPT says:  we want to. Let's just remind ourselves. We only got 3 columns in our data frame.
MattGPT says: But it would be nice actually to have that information
MattGPT says: you patching to me.
MattGPT says: There we go. It would be nice to have
MattGPT says: that information in our data frame. Okay. so what we can do is
MattGPT says: use the map function. If the map function is very nice.
MattGPT says: If you have a dictionary and you have
MattGPT says: values, the the key represents the value. So you can see here whatever uses
MattGPT says: top fibros in the candy column.
MattGPT says: Okay, so what map does is it goes road by road and says, okay, the first one is twigs.
MattGPT says: Let me get. Let me do. The equivalent of saying is chocolate. Square brackets in quotes twist, and that will give you back either 2 or 4 right
MattGPT says:  and so Matthew will take this column
MattGPT says: and
MattGPT says: take the values, row by row, and then look up. Use the vitamins in chocolate to look up people.
MattGPT says: So it gives us a way to add a new column into our data frame.
MattGPT says: Okay, so again, this is how we add a new column. a
MattGPT says: right-hand side. But stop at the data frame.
MattGPT says: Select the candy column. apply to that column the mapping function. Okay? So each value in the timely column map it using is chocolate dictionary.
MattGPT says: That means, take the value that you have in the current row. Look that up in your dictionary at the D, and then
MattGPT says: the associated value come back.
MattGPT says: and then to add a new column. We just put the name of the column on the site. So it looks like adding a new key value pair into a dictionary. You can see that when we do that we now have 4 columns in our data frame
MattGPT says: added in using that money, because that's a really common thing to do and especially when you come across data. which is data you want to work with, but it may be encoded, you know.
MattGPT says: like that, right?
MattGPT says: You don't always have to. You first have to make sure that you understand which what Durham actually what one map is.
MattGPT says: but it would be better than to have like. Then the label up, I think. where you found out that 0 was female.
MattGPT says: Then you can do that mapping like that right, because then it's much better for what we've labeled and when you do plotting better to have labels, the labels come out when you do value count. So it's getting the number of times 0 occurs number time one from this is really common to see this in data will come across, whether categorical data.
MattGPT says: phenomenal and ordinal data, but it's been encoded numerically.
MattGPT says: And this is a really common thing to do. There are some reasons to do it, but the difficult procedures like progression behind. But I think most of the time. So this is where you do a manage.
MattGPT says: We dictate and the map and say, everyone is a 0
MattGPT says: email. What? However, you want to encode it?
MattGPT says: Okay?
MattGPT says: And then they are data frame here with pool.
MattGPT says: My data data brain.
MattGPT says: Okay? Then we could pull out. stand upon.
MattGPT says: That's a dog map.
MattGPT says: Go at the end of that.
MattGPT says: And so that's a really
MattGPT says: common thing. Want to do. Add new columns is to use mapping.
MattGPT says: Okay? And then the next thing that we do remember, we were doing this by counting out the number of a different
MattGPT says: candies. Overall. All right.
MattGPT says: because each candy in this data set it was the top ranked candy by each state.
MattGPT says: Well, in pandemic, it's much more straightforward, right? So again. This is something that you hopefully your site can see already you read from left to right.
MattGPT says: And so if I give you like data, frame, select this column or first group. By this one or 2 columns.
MattGPT says: create subsets of data, then select the column of interest, and then apply the function
MattGPT says: going left by right, from left to right.
MattGPT says: and doing things will change where you're saying. But period, then you have a function. So you read this. I start with the handy data frame. select and column square brackets there
MattGPT says: and then. That's kind of horrible data to apply. The value counts. What does that do that goes through and looks at each unique value.
MattGPT says: and then the number of times that occurs, the number of rooms that value occurs in
MattGPT says: and by default it will give it to you. you know, in a ranking table from most frequent to least frequent
MattGPT says:  there's a few extra things
MattGPT says: that you'll sometimes see
MattGPT says: if you wanted to have proportions instead of discounts.
MattGPT says: You can say normalized
MattGPT says: and give that the value. And you can see the base left saying, 13
MattGPT says: States have skittles as their top ranked fans.
MattGPT says: and so on. Okay.
MattGPT says: so you'll sometimes, you know, you'll do that sometimes.
MattGPT says: by default.
MattGPT says: we get it in
MattGPT says: frequency order. This is the most useful kind of summary to get it ranked that way, but sometimes you want it alphabetically.
MattGPT says: So what you can do is that you can then apply the sort index function.
MattGPT says: What does that mean? Well, that's and again, you can see. What we've done is that we go and step by step, start with the data frame, select the column.
MattGPT says: tabulate the values in that column.
MattGPT says: and then.so index. This is your index over here, and these are the values.
MattGPT says: Okay? So when you say full index that will resort or reorder this table
MattGPT says: alphabetically.
MattGPT says: okay, this is less useful in some ways.
MattGPT says: Numbers jump all over the place, but
MattGPT says: often, you know, say, you've got the States. You may want them. You may want them in a table, or then pass this onto a chart.
MattGPT says: You show those in alphabetical order right?
MattGPT says: So you want to be careful not to get. Let these get too long, because you can keep going. Drop this drop that, you know, and adding extra functions on there. But it's not unusual to see 3 or 4 things in a chain
MattGPT says: of function calls and pandas right? And so. Sitcom. And the order in which you do them in is important.
MattGPT says: Okay? Because if we say candy dia candidia
MattGPT says: web records. Can you sort index, but value counts. That would give us a very different output.
MattGPT says: because what we're doing is that we're sorting the index.
MattGPT says: We're restoring the rows before we're counting up the values. Okay? So depending on what you're trying to do, the order is important there. and then we can always add in.
MattGPT says: and sometimes we we will see this, even within this is gonna be 2. Gonna be ugly.
MattGPT says: but you could then add a bar chart in at the end. Right? So you've got that table.
MattGPT says: hey?
MattGPT says: Yeah. So it's put it in  and that that could be an interesting sort of exploratory visualization there.
MattGPT says: And remember, this is just basic the basic chart types.
MattGPT says: This bar which gives you a bar drop in the standard way with the categories on the X axis.
MattGPT says: If you say File H, and you're going to get a horizontal wide chart. which is usually better for rankings.
MattGPT says: The one thing which is a little odd, that is, that it actually
MattGPT says: goes from the bottom up. I mean, that makes sense right. But if you're thinking of a ranking table where you ordered it in some way.
MattGPT says: Then it goes in the reverse how you might expect it.
MattGPT says: Okay, any questions so far on that. Hopefully, this is just review. But like, as I said before, I wanna keep kind of going over this again and again.
MattGPT says: so hopefully, it becomes sort of second nature here.
MattGPT says: then,
MattGPT says: when we want to subset a data frame. Okay, there's 2 ways in which we can do that. Obviously, we can subset by the road and say, you know, one road where the State for this specific state, just for California, just for Pennsylvania.
MattGPT says: And we can also stop set by the columns. Okay? And so we've we've seen this picking out one column, we can pick out a pair of columns.
MattGPT says: So sometimes you, you know, it's quite useful to do it this way is to make this up a list. Say, I just want to get some people
MattGPT says: from the data framework sub data framework. I've just got the state and the amount following. okay? And so you put those into a list and then handy on score. DM, and square brackets means that we're doing
MattGPT says: indexing.
MattGPT says: We're doing column indexing. And you can see that we just we get the whole. We get all of the rows, all 51 rows.
MattGPT says: But just those 2 columns. Okay, so that's an example of
MattGPT says: column subseting. Create a list
MattGPT says: and then pass that. But more commonly, what we're doing is actually subseting the roads based on some value.
MattGPT says: And so this is where we use the idea of a road filter, which is
MattGPT says: conceptually the same thing as what we were doing with using
MattGPT says: a a conditional, a loop and a condition. When we were working with basic
MattGPT says: loop over each of the rows for each item.
MattGPT says: apply a condition where we have a test. If if it meets that test, it's true, then do something with it. It's an idiot that you had in assignment 2 and one was that loop and filter sort of idea. Start with an empty list, chocolate candy loop over all of the candies, check whether the candy is chocolate or not. If it is.
MattGPT says: add, in the name of the candy to the chocolate
MattGPT says:  Dependency approaches a little bit more direct. What you do is you construct
MattGPT says: a roadbuilder? Okay? So in this case. in our data here, we're looking just for states where the amount associated with that State is greater than a hundred 1,000.
MattGPT says: Okay, so what we do here for the final filter and here select column of interest. And then we apply the test
MattGPT says: great to Numera, so we can say greater than 100,000. That will give us back 51, a list of 51 true or false values.
MattGPT says: And then we can apply that. Okay, so just to reinforce what we're getting here.
MattGPT says: A,
MattGPT says: that's okay.
MattGPT says: Okay. So for each road. this is the result of asking this question is the amount
MattGPT says: greater than 100,000? No, it's not so, that's false. No, it's not. That's false. Yes, it is so, that's true. So we should have true, true, false
MattGPT says: continues there. So every time you define the row filter on a specific data frame, you'll get back a list of true or false values, the same number as the number of rows that you have.
MattGPT says: Okay? So that's what a row filter looks like.
MattGPT says: or sometimes it's really good to combine to create multiple filters. So I want states that begin with the letter A, and where the amount is greater than 50,000,
MattGPT says: and then, when you use them in select, you can combine them
MattGPT says: in a row, select
MattGPT says: or a road built in here. So the way you do that again is the name of the data frame.
MattGPT says: Yeah, right, is it? And now we're giving it all those guilty.
MattGPT says: Remember, our roadfield risk is the lift of true or false values. Where it's true, we'll get back that road.
MattGPT says: Okay? So you can see what we've done is what we've subset on data range. Just a road that
MattGPT says: where the amount was Beta, 900 pounds.
MattGPT says: Okay, so again, this is a much more direct way than we were working with. You have to write a loop.
MattGPT says: You don't have to directly write an if condition
MattGPT says: you just have to think about, what is it that I'm trying to do? So it's much more like looking at your data and saying.
MattGPT says: Okay, I want to get a subset of rows based on what based on values in this column. So I'm going to define a row fielder by selecting that column and applying to test
MattGPT says: or it may be that it's a combination of the cost.
MattGPT says: so they can. They can get more complex in that way.
MattGPT says: Okay, any questions on filters, row filters in particular. or you want me to explain it in a different way.
MattGPT says: Okay, again, hope, I'm hoping that this is just like you like, oh, yeah, I got this already. But I just wanted to be
MattGPT says: to go over it again and give you this this notebook in a different form, so that you know, a different data set. Hopefully, this makes sense. Another common thing is that we've got a data frame by default. It will be in the order in which it is in the file that we load it from.
MattGPT says: but often actually a quick thing that we might want to do with data. Enough that
MattGPT says: before doing any filtering is, we may just want to restart the data frame like reorder the rows based on a particular column. Okay? So the sort values function gives us that
MattGPT says: abilities to do that right.
MattGPT says: And so again, we just say, start with the whole data frame. That means we're applying a function to it full value.
MattGPT says: which takes one or more type of argument or handling a number of arguments. Take a list of problems. So here we're saying one problem here.
MattGPT says: and by default, for numeric column and for our alphabetic. You know character path. It goes from smallest to lowest. So be an alphabetical order from a up. We did that one state
MattGPT says: for a mountain. It's up to the smaller.
MattGPT says: So it's going in ascending order there so that can be really quick and useful way to look at our data. And again, you can notice the
MattGPT says: ruined, that tiff. Remember, this started from 0 through one in the order that is noted in
MattGPT says: that keeps it there sort of as a a reference.
MattGPT says: The State with the least spent the least amount on candy was Connecticut.
MattGPT says: That spent the most is California. Okay? If we want it in in reverse. and we can apply the ascending a.
MattGPT says: the ascending argument to well, by default. That's true. But if you say Oh, they're not. there isn't an argument which is descending equals. True.
MattGPT says: the era sending was true host.
MattGPT says: and so we could go from the largest, smallest. so that can often be a really easy way just to answer some quick questions before you even do any filtering right? So say you want to say, what are the top 5
MattGPT says: States in terms of spending on candy?
MattGPT says: Then this is the Panda sort of idiom. Again, start with the data frame by the first function, sort it from most and then subset it. Or just look at the top 5 rows.
MattGPT says: Okay. and then the least would be the other way around, wouldn't it? Right? So
MattGPT says: you can just set that to true. So it is going from smallest to largest.
MattGPT says: And then you've got the
MattGPT says: the top, the the
MattGPT says: the top 5 lease spenders, I guess, or the the bottom of the list there.
MattGPT says: Okay, so that's a really quick and easy way to to do stuff. You can combine up columns in there.
MattGPT says: And so then this last bit here shows that we can have
MattGPT says: 2 or more levels of sorts. Okay, so let's say, we sort
MattGPT says: the data frame by the candy columns.
MattGPT says: I am okay. So here you can see this alphabetical. So we get B
MattGPT says: be out for bending
MattGPT says: right this is a good way to sort of group them together. And so you can see, like the number of states that have candy corners, their top one
MattGPT says:  and
MattGPT says: but we can also do this
MattGPT says: in sort values. If you pass it a list of columns. It will sort first by this column and then by this part.
MattGPT says: So you can have multiple levels of sort in there.
MattGPT says: So this is going to sort both of those alphabetically.
MattGPT says: And so you can see first
MattGPT says: candy corn.
MattGPT says: We have 5 candy bones. Until then it goes to the second level of sorting
MattGPT says: so that we're getting it ascending order. IA id nd.
MattGPT says: right?
MattGPT says: So a lot of the questions that you can ask answer quickly will just be signing your data frame and saying like, let me rearrange it first, so I can understand what's going on. And I can answer quick questions like. What's the top.
MattGPT says: you know. What's the top value for this? What's the top value for this? Or what's the top 5 plus the
MattGPT says: you know? What were the 5 countries with the largest
MattGPT says: Gdp, what are the 5 countries with the least amount of emissions, you know? CO. 2 emissions, or something like that?
MattGPT says: Okay, so that hopefully is reviewed any questions on that.
MattGPT says: Okay? So we're just going to practice some practice and purchase with this
MattGPT says: and adding in some more things with that on top of that, and how we work with panel.
MattGPT says: Okay, so
MattGPT says: second notebook here.
MattGPT says: we're not gonna go over. But this again. And I wanted to give this to you as practice. Okay, so this is our baby names data frame file that we have looked at before.
MattGPT says: Okay, so this just gives you this. That's again loading in the data.
MattGPT says: and it reviews all those things here you can work through and run all of these, and then you'll see there's some blanks in here
MattGPT says: to fill in these. How do you
MattGPT says: do? Do various ones of those. So this is an assignment. But it's just like more practice
MattGPT says: for you to do that. Okay, and there's some tasks here.
MattGPT says: and I would, I would suggest, as you're practicing this is find itself another data set. Maybe you have one that you're interested in. You just happen to have one on your computer or from another class.
MattGPT says: Start with a relatively simple one way. You've got only 5 or 6 columns that you understand, and maybe 50 or 100 rows. But then you can move on to more complicated data sets.
MattGPT says: Okay? So the rest of time, I just want to
MattGPT says: go over a more complicated example.
MattGPT says: So you remember in Chapter 3, the foundations, Jones talked about 3 domains of data around it, data because he's got this cute thing, the chapter number, and the number of
MattGPT says: books about him.
MattGPT says: And this is the foundation of how we've got our assignment States right? So we've got public.
MattGPT says: private and personal data realm. And so the public domain of data
MattGPT says: refers to data which is
MattGPT says: publicly available. Right? And a key example of that is what is called open data.
MattGPT says: And it's Tom, open data. We looked at a little bit, a few quizzes. Back. You went to open data, Philly, and you looked around at.
MattGPT says: some of the different data sets sets that were there. And you picked one and thought about like, what? What does it describe? What kind of questions could you use it to answer? Okay.
MattGPT says: but you'll find that these that this term open data is a little bit broader than just meaning
MattGPT says: open data in like data. Dot Govs is like the
MattGPT says: national site for governmental data. And then most cities have at some point got their own data portal. So we've got open data, Philly, that
MattGPT says: and Nyc data, and so on.
MattGPT says: But this idea of open data is is used in other contexts, and particularly in sort of industry
MattGPT says: and organizing different organizations in slightly different ways. Okay? And so
MattGPT says: in a business or enterprise context, you may come across talk of the importance of making the data sources open.
MattGPT says: Okay? So even within a huge company.
MattGPT says: you often find that people don't have access to data
MattGPT says: across the whole
MattGPT says: spectrum, or of the whole organization.
MattGPT says: And so there's been a move over the past or 20 years. Try and move towards opening that up, you know. So you may be in the marketing division or sub part of the market division.
MattGPT says: You don't have access to sales data.
MattGPT says: or, traditionally, haven't
MattGPT says: to to. For within organizations, to sort of open up their data and to have data repository so that people can work across those
MattGPT says: the key thing there is that to what we're trying to do with data? Science is the
MattGPT says: find. Admirable insights. you know, insights and conclusions from our data.
MattGPT says: So that we can then move up the data hierarchy up to some knowledge that we gain. Apply that to make change, ie. Wisdom.
MattGPT says: And so within the business and enterprise context. Then you'll hear talk of open data.
MattGPT says: Which is different to another place that you've seen this whole idea here of open data at various times. And it's become somewhat controversial
MattGPT says: is
MattGPT says: various data activist initiatives. Okay, so we've got something like Wikileaks.
MattGPT says: being perhaps the most controversial. There's other sort of hacker data activist sites where they do this.
MattGPT says: where someone has argued that there is some data which really should be in the public domain. But isn't
MattGPT says: okay? And so then it somehow managed to get it, either nefariously by hacking or just from fact that it's, you know, available to someone within an organization like when Meta did their survey about Instagram and teenagers. It was
MattGPT says: It was somebody within Meta, one of the researchers who actually released that report and some of the data there to show what their findings were.
MattGPT says:  and so you know much. Much of these data are private and then have been leaked.
MattGPT says: but they're not necessarily anonymous, and so, you know, there's some.
MattGPT says: The meadow may not be
MattGPT says: good reasons, or may be legally justified. May or may not be legally justified.
MattGPT says: But when we talk about open data, for when I'm talking about open data here, we're more talking about information from or in the public sector, in the government initiative.
MattGPT says: Take the the salaries of people who work in the police department, in the government.
MattGPT says: and local government our politicians. etc., etc. Right? that is publicly accessible. Data purpose is publicly funded.
MattGPT says: How much money people get who gets money for grants the constantly, and Nih or the and NIA. Or, you know, etc. That's publicly available data.
MattGPT says: the Philadelphia City budget year by year, and how it's broken down and spent
MattGPT says: where that spending goes. That's public Daylush.
MattGPT says: Okay? Because it comes from taxes.
MattGPT says: etc. So you can think of all of those things there. So there's been a sort of initiative over the past 25 years
MattGPT says: to say, well. you know that data has always theoretically been all of our data.
MattGPT says: and we should be able to have access to it. But to get access to it, you'd have to really work with the government or go to someone in the government and say, I want to get access to that data, and they'll be like, Oh, I don't even know how to give it to you.
MattGPT says: This is in some database that I don't really know how to access. and so on. So sometimes there are the, you know that, and and I probably say the majority of times data is there. but not available. And that's not necessarily because someone is trying to obstruct the data from the public who should have access to it.
MattGPT says: It's just that people don't really know the best way to share it.
MattGPT says: And their job is to sort of
MattGPT says: work in a specific department specific area with those data and may produce a report with those data.
MattGPT says: but actually to go to the next step of knowing how to prepare those data so they can be made available to other people.
MattGPT says: That's for a lot of people really beyond where their skill set is.
MattGPT says: But the whole idea behind open data is that data that can be free to use, reused and redistributed by anyone? And this idea of
MattGPT says: attribute where you've got the data from
MattGPT says: and say, Well, you know, I've I've worked with these data. I got it from here. I've added in some columns, or I went through, and I cleaned it up. And here you can have then access.
MattGPT says: And so the key elements
MattGPT says: in this kind of understanding of open data
MattGPT says: is that the data should be published in an accessible machine. Readable format.
MattGPT says: Okay? So it's not just, oh, yeah, you know, you can go to the library and get it
MattGPT says: principle, then you have to re-enter it. If you wanted to actually
MattGPT says: start to analyze it.
MattGPT says:  and it should ideally be in a machine readable form like a Csv file, excel, file and not a Pdf, I mean, you can extract data out of Pdf.
MattGPT says: but ideally, the goal is to have data available in
MattGPT says: the easiest machinery of reform.
MattGPT says: And then it's there available for anyone to use. Okay? So you should just be able to down, go to a site and download it.
MattGPT says:  and then an additional tool with that is that you put a license on that that says that open access.
MattGPT says: So
MattGPT says: yeah, the key features, I think, are availability and access should have another C in there.
MattGPT says: reuse some redistribution and universal.
MattGPT says: And this is the the dream or the gold.
MattGPT says: and why we wanted to become more data literate and have these skills is so that we can actually use those skills.
MattGPT says: For you know, using data for good. Okay? And that can be in our personal, private, or in this public realm where we're like, okay? So in my community. we're trying to improve
MattGPT says: the air quality. Is there some data available about the sources of pollution.
MattGPT says: Or you know the temperature over time. We've already decided to see those data, plus where trees upon it, or how many trees there are. You could piece together all of those bits of public data and make a data argument, database argument for
MattGPT says: trying to bring about change. Okay. So alongside doing important things like.
MattGPT says: you know, protesting or writing to.
MattGPT says: you know, representatives and so on. You can also try and bring about change through data activation.
MattGPT says: and so, but you have to have it ideally. You should be able to get access to those data
MattGPT says:  this is the home of the US. Government open data.
MattGPT says: And you can see
MattGPT says: there's a lot of data out there, right? The quarter of a million
MattGPT says: data sets.
MattGPT says: and you can, you can browse this and look through by organization, or
MattGPT says: most viewed or there's other ways in which you can look at that. So I'd encourage you to to spend some time looking at this.
MattGPT says:  What you'll find
MattGPT says: is is really patchy in terms of the kinds of the things about there, and sometimes. The period of time that covers because and it's kind of bursty terms of initiatives in open data.
MattGPT says: where.
MattGPT says: under the Obama, 2 of Obama administrations, a lot of this stuff really kind of had a boost right, and he introduced. The first
MattGPT says: chief data scientist who was actually. You know.
MattGPT says: had a had a specific role to support and push these things forward.
MattGPT says: one of the first things that Trump did when he got in.
MattGPT says: We'll stop all of those open data initiatives. And so you can actually see when I get to 2016, we finally have a drop off in some of the availability of that data. All of that data is is constantly being generated. Those reports are being generated within the government. But that last step of saying, Okay, it's important
MattGPT says: to take these reports. Put them in such a format and publish them
MattGPT says: so that anyone can get access to them. So you can still get hold of that data. But it's not necessarily through these portals. Always.
MattGPT says: so that's data. Gov, And then you've already seen this one data. open data, Philly.
MattGPT says: a lot of really good stuff there. But again, you'll notice when you look at it. It's kind of patchy and the timeframes are covered. Some of the data sets
MattGPT says: Philadelphia police Department one that we're gonna look at. The shooting incident that is constantly updated almost on a daily basis. Great
MattGPT says: the crash database.
MattGPT says: which I think we looked at.
MattGPT says: you know, for vehicle
MattGPT says: collisions and accidents that one is updated.
MattGPT says: But some of these things, where was really interesting, I thought, where are the recreation? Thanks. We're in Philadelphia.
MattGPT says: Where are the trees planted as even environment
MattGPT says: where the potholes are.
MattGPT says: where there are abandoned lots, there's already where the street lights are that sort of stuff.
MattGPT says: Those can be a little bit patchy. So sometimes, when you're trying to answer a data question, you can say like, Oh, I've only got that up until 2018. Oh.
MattGPT says: which is a shame, but it gives you something to start working with. Okay.  this. Another example here from Maryland.
MattGPT says: kind of things to look at.
MattGPT says: Spend some time with U.S.A. Fax is an interesting one. It's kind of developing
MattGPT says: So
MattGPT says: this is a nonprofit that was started by Steve Bauer, who used to be after Bill Gates was the CEO of Microsoft for a while. and I.
MattGPT says: Everyone
MattGPT says: operator.
MattGPT says: So it's
MattGPT says: and there's another one.
MattGPT says: The source of
MattGPT says: yeah, I, personally.
MattGPT says: okay, Hi, everyone doing okay
MattGPT says:  high temperature data is being like a new mobile, for I mean our favorite.
MattGPT says: Okay, cool.
MattGPT says: I work through the assignment.
MattGPT says: and they get through. Pull up! Get through.
MattGPT says: I'll let you know when 2 is there about another 5 or 6 people to go
MattGPT says:  overall? I think everyone is doing great and hope that you're finding this a good way from, you know, the exact that we talked about last week like a way to
MattGPT says: best learn this stuff is food
MattGPT says: products of work groups.
MattGPT says:  and so I hope that you're finally the assignment.
MattGPT says: stretching you a little bit, but also not picking you out and out, and but
MattGPT says: but hang in there. So I have one more this week.
MattGPT says: 4, which is based on using kind of on open data.
MattGPT says: And then next week
MattGPT says: is full break week at the end, right? Or Wednesday, whenever it is instead of what would be an assignment next week.
MattGPT says:  And next Tuesday.
MattGPT says: That will be a 5,000. But it will be on the minimum.
MattGPT says: Okay, that's fine.
MattGPT says: what I want to do because we're spending week 7 is basically use that session to review
MattGPT says: one of the kind of the basics and the data analysis
MattGPT says: for those of you that haven't done this as much stuff before finding that we're moving, because too quickly, or you just wanna
MattGPT says: go over the over the what was covered
MattGPT says: and use that as a review session, and when we come back off the break
MattGPT says: we'll cover some of this in public sort of things.
MattGPT says: I think that we've just. I mean, we want to keep on working through pandas
MattGPT says: but I wanna cover Geoloc, Geo mapping and doing geopanders geographical, following Middleton and geographical
MattGPT says: data analysis. What you think
MattGPT says: geolocated incidental points.
MattGPT says: and you can map them in a different
MattGPT says: shapes. Basic codes. And we've got where paragraph is.
MattGPT says: were individuals
MattGPT says: crashes into that, or you can say, let's do that at the block level, or let's do that at the whatever the unit may be.
MattGPT says: So I think we've already spent a couple of weeks in that and then
MattGPT says: and it's a way of doing in practice costs. So works just like map in lots of ways.
MattGPT says:  but you get automatic map tips or map over onto point. So it's really good for data explorations
MattGPT says: halfway, moving towards a dashboard
MattGPT says: like like.
MattGPT says: So then
MattGPT says: there are a couple of things, couple of options that we can
MattGPT says: kind of built on top of handed
MattGPT says: to produce dashboard. And if I want to take a look at a couple of those ones called Streamlett, and then one that
MattGPT says: with the Pandas basics. The title basics we have, plus what we've got in pandas to be able to
MattGPT says: play around with those then data story,
MattGPT says: having dashboard seen as
MattGPT says: the path that's coming our data forward, right?
MattGPT says: but I think that leads a lot of people either can work in certain contexts where you're trying to present data for a team
MattGPT says: where they also want to explore the data.
MattGPT says:  but
MattGPT says: you know whether I'm pushed out to the public as a data story. Then I think but I wanna show you how
MattGPT says: that
MattGPT says: great stuff  And then, if people have.
MattGPT says: you know, people have ideas about stuff they'd like to cover. I'm open to our that do? how much impact might there be in some of work work-saving?
MattGPT says: You think that's in my business. So cool? Alright. So I'll go at that end and stuff
MattGPT says: this that can happen at different levels.
MattGPT says: This is the very simplest way you've got, like Wikipedia, where there are people of data, for various things, populations like
MattGPT says: world population, or anything which is put up there. And people.
MattGPT says: you can actually prompt pandas.
MattGPT says: use a function which is read HTML, and that will load in all of the pages of that page into the data.
MattGPT says: 5 years.
MattGPT says: So that's a really easy way to do it. But often you'll find that it's
MattGPT says: somewhere on the web, in the table or in some other format.
MattGPT says: a little bit more involved screen. So that's kind of like real. There's like 2 or 3, and often for doing that. So
MattGPT says: we'll go some of that stuff and see
MattGPT says: where we get to that. Okay,
MattGPT says: cool. Alright. So if you want to fetch Number 10, August 10 today.
MattGPT says: what? We're gonna be working pretty
MattGPT says:  So the first one
MattGPT says: it's just the next part of.
MattGPT says: remember this typology of ways of using data and data storage. It comes with this article. It's one way, there's a debate
MattGPT says: things. And so we had looked at those basic ones. So, thinking back to what we were working on last week, where we're looking at the rap
MattGPT says: sightings, or the cool, the cools about rep sidings the 3 1, one and Neil
MattGPT says:  We were
MattGPT says: in a specific year in a specific district, you know. So in Manhattan, but in Brooklyn, but queens
MattGPT says: in a physical year, right? And so those kinds of questions.
MattGPT says: measurement-based preference. They have a measurement based on. See the way that we do that by filtering or in pandas or value counts where we might do 2 levels of value counts, or we might say, group buy and then value count
MattGPT says: that would be able to see, like a number of findings.
MattGPT says: the environment, probably different years, and so on. So that's just simple. Many members of making how far you can get enough in telling a data story
MattGPT says: and presenting that by this working environment. And they're proportionate with staying that entirely.
MattGPT says: Okay. So out of all the rest replying, and 2021 what proportion were in each of our and that would end. It's up to maybe
MattGPT says: visualization. That's that botany. Not a bike up
MattGPT says: anyway? And then the other 2 that we don't have a sort of comparison where you want to compare between
MattGPT says: do same things within a data set a cross state of that for 2 different editors.
MattGPT says: So no, I think covered those. And we've got the basic idea, when we call our data frames
MattGPT says: to to build queries in patterns right? And we really do want to think of it in that way by this kind of idea of moving from left to right for the start
MattGPT says: with our talk about data brains. But by what that is.
MattGPT says: and then we may
MattGPT says: apply field documents
MattGPT says: I do not need. So with some credit, our data there
MattGPT says: and then, we may then want to know what's the average number of
MattGPT says: variable one.
MattGPT says: what we should be seeing and getting comfortable with, and
MattGPT says: a lot of this stuff you're still thinking in the same set. But you don't have to be. You don't have to necessarily physically, never have to be playing contact. But out of all my roads
MattGPT says: let me apply this filtering pad something like in my data frame.
MattGPT says: Yeah.
MattGPT says: 99, right. I mean, if that is up. let me pull that like wide bill or something.
MattGPT says: hey? So we think through like, what do I do? First, the ordering here is important on having
MattGPT says: select, but once you've filtered the data, then select one or more columns
MattGPT says: to then apply some sort of content to and then
MattGPT says: depending, you know, you may then like again, I don't know what. So
MattGPT says: okay, but we we move across in that way,
MattGPT says: and then often, you know. So what we do, we
MattGPT says: we can do filtering at this point right here, and for that will come before we do the selection of the variables and the fourth function, and then
MattGPT says: categorical, variable. And we want to split our data frame up, start to stay the same. Why is the values in that variable?
MattGPT says: And that's where we've been and money back to retire the group by from 10. Take an argument. This is one more problem.
MattGPT says: We'll then slide the data, frame up that way or group. It
MattGPT says: rearranged itself, but based on what it is but category variable. One.
MattGPT says: Say, that's got 5 values every regions event in the world, or something or regions in Florida, in New York City. But then we get Manhattan Queen.
MattGPT says: the prompt. Okay? And then we can say, now, let's get variable one
MattGPT says:  and than a pipeline. Okay?
MattGPT says: Again, the way that it's going that way.
MattGPT says: So that's how what we're thinking about. But sometimes I mean, you don't always want to have these massive one-liners that get way.
MattGPT says: you know, way myself is like. Okay, I wanna create filters I would always find on a special line. But it's tend to reuse filters.
MattGPT says: Some people like they're trying to define them. And you'll see this.
MattGPT says: Put it in 5 there. So you could say.
MattGPT says: Rainbow.
MattGPT says: yeah, 4, 99,
MattGPT says: and then do your things in there. So you're putting a filter kind of physically in there.
MattGPT says: I hadn't shot away from those. I think it's too complicated in a way. Give them a good name.
MattGPT says: and then you can find them, and if you have more than one builder, so you have the year builder, and then you might also have a categorical builder of some kind
MattGPT says: to to data, frame category, one
MattGPT says: wrong or something
MattGPT says: then you can combine them. They can say
MattGPT says: all of the instances, all the sightings of wrapped in 1990 what they are data. And in the box, right? And then you can combine those
MattGPT says: get back into.
MattGPT says: So that's kind of your assignment.
MattGPT says: I think I said it sometimes when you choose. Name. Okay, I know it's like, you know, something in my list.
MattGPT says: But even here more play for my builders.
MattGPT says: So you know, go to the other extreme and have really really long district response. But try, make it meaningful right? IJ. Kx, doesn't mean anything right? So 4, I in, you know, that's a very common programming video. But what we're trying to do with data programming and data programming make our codes readable as well.
MattGPT says: One of the really cool things about contact, I think, is that it actually reads really well, and some of that has to do with invitation but also keywords like. For all these other things, they almost read out Microsoft, right? So if you have.
MattGPT says: you think about the names that you give to your point of your object
MattGPT says: that help choose. You remember what you brought me, but even if someone doesn't know the pipe and then they can read through. They can be like, Hey, I can figure out what's going on there. But if you just, you know, I in
MattGPT says: brain with range working birthday land
MattGPT says: this and left. You know, it's it's kind of agreed as well. Kind of
MattGPT says: yeah, and that, I think, is it a full goal?
MattGPT says: Okay? So in this. Then next to the cover here, it's just way about change over time.
MattGPT says: and the idea of being tabled right. And so
MattGPT says: you you can take a look at these, I think, really. But let's just
MattGPT says: run all of these.  you remember, we went through these 5 data sets that we were looking at  we're actually doing the load in inspect
MattGPT says: observations. That's just the same from part one A,
MattGPT says: the
MattGPT says: alright.
MattGPT says: We need a bike in the ballpilment, right? So bike.
MattGPT says: So
MattGPT says: the one thing I don't said that you probably discovered it. But like you've put in a notebook.
MattGPT says: have had conditions, right? So just
MattGPT says: once included a data frame.
MattGPT says: You've got a lot of columns in it, and you do something like this open rabbit and stuff. But, Vi, I can't remember what that column is called. If you press tab at that point, then you'll get just like a regular sort of
MattGPT says: I'll see you if you use that
MattGPT says: other ids digital code.
MattGPT says: And so that would help build it. For
MattGPT says: like enrolled.
MattGPT says:  okay, so if we're looking at the number of bike crashes and data set that we had over time.
MattGPT says: That would be the feature, that is, that we can first subset our data frame
MattGPT says: so that we only have roads where the bicycle count was greater than 0. It's left our filter right there. Bike involved.
MattGPT says: This is an example where you might. You could do this sort of one on on one line. But because you might use your craft.
MattGPT says: the device will crash data, frame multiple times. It makes sense to do the filtering. The road building, first
MattGPT says: on the whole craft data frame, and then call it something like.
MattGPT says: And then you see that we can suck that in the data frame there so that we go
MattGPT says: just the crashes. The crash incidents where the bicycle cameras were able to do it.
MattGPT says: Okay.
MattGPT says: we could just do value counts. Crash here. That'll give us the same information.
MattGPT says:  but you can also use group by as a way of creating the subset. And then an adult size function would count up the number of roles.
MattGPT says: Okay, so this gives us change over time.
MattGPT says:  and see, we could just look at that as
MattGPT says: visually. Inspect that and say, Well, there is some change there, but it's all within that group.
MattGPT says: Pop up
MattGPT says: But again, here, once we have
MattGPT says: table like this by default. We have, if we just say 12, and we select the buffer.
MattGPT says: then
MattGPT says: the index, which is the left-hand side. Here again in this table, here the crash year will become our practice, and the value
MattGPT says: will repound the height of those bottoms.
MattGPT says: So
MattGPT says: looking at the end of the time
MattGPT says: so we can see. Yeah, there's some for a pattern. There was a
MattGPT says: like increase dying in 2012,
MattGPT says: and then
MattGPT says: the
MattGPT says: then sometimes that's how we would do things every time. Okay, so the basic principle you can go across the other day is that
MattGPT says: but we don't find involved. Find out what your ear variable is
MattGPT says:  group by that variable, and then and do whatever operation you can get
MattGPT says: could be a need in this case, just the number of it. So we
MattGPT says: 5 on
MattGPT says: and then ramp tables. Same idea here. But what we're doing is is ordering data according to the measurement, provided comparison
MattGPT says: of more than a handful of value. Okay, so.
MattGPT says: and we had the child led blood level.
MattGPT says:  okay, yeah. So this was on table for the prior.
MattGPT says: So
MattGPT says: we might want to do something like this is the numbers we
MattGPT says: okay? And so
MattGPT says: and then this is the average and
MattGPT says: number of but pop the bit in
MattGPT says: greater than 5 plus okay out of that number. So we might want to work, make work out 14 out of this, but divided by list.
MattGPT says: And so
MattGPT says: do that. Then?
MattGPT says:  okay. So we've had a new call and then
MattGPT says: alright, check up my name.
MattGPT says: Okay? So we're gonna order. We can use sort values
MattGPT says: the way. And often this will answer a lot of our questions if we got data, full value, full value one on all columns. And so this is a single column. You give it a string. You want a series of columns, first pull by the first screen, and then go by something
MattGPT says: and give it a lift there.
MattGPT says: And then, as I'm sending value by default, it will be from
MattGPT says: more of the largest if we want it in defending order. Then we can set spending.
MattGPT says: Okay? So that's what we've got there.
MattGPT says: so that gives us the ranked right as simple as that. Just by using talk values.
MattGPT says: particular variables. You want to rank by
MattGPT says: pumpkin away. And
MattGPT says:  remember, we needed to
MattGPT says: per month. So they're gonna be 0 1,
MattGPT says: 2, 4.
MattGPT says: Okay? So again, here, what we're doing is we've got region well
MattGPT says: in the region code here. So that's our first step. What we want to do is create a ranked table.
MattGPT says: All
MattGPT says: more into the larger place. There's there's
MattGPT says: the across the year.
MattGPT says:  total of the anticipation.
MattGPT says: Okay, so this is released. Pieces of metals to the most pieces of mendels by reading what we're getting at.
MattGPT says: and that's another example of what would be considered a rank table.  and
MattGPT says: Ms bluff, Ms bluff is novel and
MattGPT says: change. Okay, so where you've got that data set where we had urban. Unfortunately, urban versus rural
MattGPT says: projected
MattGPT says: what we want to do is create a change variable. Okay? So
MattGPT says: for each.
MattGPT says: for each country.
MattGPT says: we want to see the proportion in 1950 compared to the projected proportion in 20.
MattGPT says: So I could do that.
MattGPT says:  there's different ways, and we can do that. But here, this just shows you subsetting. So let's just take all of the countries for 1950, the values for 1950 and the 2050.
MattGPT says: And we're gonna put those into 2 separate data frames.
MattGPT says: And then we're going to merge them together. You have 2 tables that have common
MattGPT says: value, and and numbering the country
MattGPT says: and so what we do is we merge them together. So we have 2 country and 1950 and
MattGPT says: with 2 columns, the 1950 and the 20.
MattGPT says: Okay,
MattGPT says: And so once we have a table like that, it's really easy to do a calculation which is locked in blue
MattGPT says: percent change between that page 2050, we take 1950 away to 2050, and that will be a percent change.
MattGPT says: And
MattGPT says: different column.
MattGPT says: And then we can sort by that difference
MattGPT says: again, go either way
MattGPT says: from smallest, the largest up from large.
MattGPT says: Okay, so projected. These countries are actually projected to get less urban by 2050 now than they over leaves
MattGPT says: gig above view
MattGPT says: through that
MattGPT says: minus 3.8 and then some of
MattGPT says: saying a hundred percent like the room is already 100% urban
MattGPT says: Thailand, and so on. Okay, so we can, we can see how
MattGPT says: how that works right? Or we could go in in the opposite order
MattGPT says: from the largest moment need to be. The countries themselves have the highest rates of urbanization.
MattGPT says: 550.
MattGPT says: Okay, so that number you can have a look over. I'll post again that fixed one. Sorry for the couple of errors that I think.
MattGPT says: that's how that works.
MattGPT says: Okay? And then, before we actually do some parts together.
MattGPT says: Just let's have a look at second one
MattGPT says: which gives us some useful content. The data meaning of budget. Okay, so
MattGPT says:  again, we're working with some of the same data here.
MattGPT says: which is
MattGPT says: the Mebo day, is that?
MattGPT says: Well, that's loading.
MattGPT says: we talk about this a little bit more, and so for.
MattGPT says: But we'll know there's contact. Why. okay? And so
MattGPT says: by table, we've got white data you would have.
MattGPT says:  okay. country.
MattGPT says: You know something about that.
MattGPT says: We just saw that with a neat data that might have some other categorical variable down here.
MattGPT says: But then, when you get to a p measurement, which is the amount of vehicle per year
MattGPT says: by month. And January. February. Wow, you have phone calls there.
MattGPT says: Okay, so this is this is considered what's wide data that where you have repetition of your indicator or variable interest, and you're measuring
MattGPT says: and repeated columns there.
MattGPT says: right? And that would be fine. Certainly. Good way to look at data. And the reason we come across data, which is why often it's because data has at some point probably come from a report, or it's been work within, excel in such a way that
MattGPT says: people can look at it. It's much easier to look at the data kind of like in a summary way here, this is white data.
MattGPT says: Okay, I'm interested in Angola. Find a year I'm interested in. And then you just.
MattGPT says: And that gives you patents. Yeah, that
MattGPT says: my, my mom?
MattGPT says:  sometimes you want to aggregate, that it becomes more difficult to aggregate those
MattGPT says: sign at full column sum across all of these columns, and then have a total there, or if you wanted to mean by month, then you would still be saying across all of.
MattGPT says: but certainly for visual inspection. And for certain types of data analysis.
MattGPT says: why data is good and final.  But often what we want to do is transform this into long form for the long form. That each one of these
MattGPT says: where you've got repeated columns, you turn it round so that you have
MattGPT says: one row per call that. So instead of here having one road, the country a, we're gonna end up with 12 roads.
MattGPT says: So our table one table is going to like
MattGPT says: country, and then whatever variable.
MattGPT says: And then we would have
MattGPT says:  alright, we'll
MattGPT says: value
MattGPT says: name. So that would probably be months and and basically legal.
MattGPT says: Okay, so country a. And in January
MattGPT says: would be 79 Pm. A. In February
MattGPT says: was the 19
MattGPT says: 37 people, and so on. So you go all the way down. So we get a December.
MattGPT says: But this is long.
MattGPT says: This is a long form.
MattGPT says: Yeah.
MattGPT says: when you look at data you sometimes want, yeah, you want to get a sense of. Is it wide, or is it wrong? It looks like this. This is like, hey? It looks like this where you've got repetition of a categorical, variable and the same thing measured, but quite different.
MattGPT says: 5 different.
MattGPT says: I mentioned them like my mom. Then that's long paper. right?
MattGPT says: so
MattGPT says: let's get our mark Com once again.
MattGPT says: Okay. if we want to go from one to the other
MattGPT says: in pandas, then there's this function called, okay. So, going from wide to long
MattGPT says: melting happening. Melt the data.
MattGPT says: hang on here. and it becomes long and the opposite going back. Long, wise pivot. Okay,
MattGPT says: But what you have to do for a melt here, you can see
MattGPT says: this is the
MattGPT says: on the right here the malfunction.
MattGPT says: you need to say which of these columns which column or column. Right here we could have some additional category over one like the Ifo code income level, or something like that
MattGPT says: which ones are those identifier columns, IE. They are going to be repeated. But they're going to do with the key variable here. economic growth.
MattGPT says: But we want those that needed again. Which of the columns of India.
MattGPT says: Actually, the variable problem that we want melted around that way.
MattGPT says: So it's the same again between
MattGPT says: variable and value here, hey?
MattGPT says: And
MattGPT says: id got which which column the with its id. Okay. So in this data frame here.
MattGPT says: 4 id columns.
MattGPT says: So we want to keep the same region, Iso, free country and year. We're actually going to have a really long data frame. If we're going to have 12 rows for every country for every year that we have in our data set.
MattGPT says: So for the melt, we're counting here as an identifying variable as well.
MattGPT says: And then our value variables where where our value is coming from. They're coming from the
MattGPT says: 5 to the end. January, December.
MattGPT says: So then, what do we want to cool? Variable from? And of course.
MattGPT says: Okay, that makes sense. So we know that we're going from this wide data. And what about the landing physical?
MattGPT says: So that we see that our original data frame. Thanks.
MattGPT says: 1,406 road and 50
MattGPT says: column. That was the full variable for id column
MattGPT says: and region. And then 12.
MattGPT says: What do we get when we melt it?
MattGPT says: Yeah, you can see now that we get around 20,952 rows.
MattGPT says: And okay.
MattGPT says: set up a 4 id ones
MattGPT says: plus a value, a variable and a value.
MattGPT says: Thank you. So that is what long form is like.
MattGPT says: Each country
MattGPT says: each year for each month.
MattGPT says: Resource it so that they go
MattGPT says: well
MattGPT says: and road is taken.
MattGPT says: Yeah. Now.
MattGPT says: okay,
MattGPT says: And so now, what is the orientation? You know what's the structure of our data.
MattGPT says: each road, the culmination of country year and month. Okay? So if we group by company
MattGPT says: you can see that we still get
MattGPT says: 9 years of data in there. Okay, in our original data, right? We had 9 rows because there were 9 years
MattGPT says: 12 in each row. We had 12 variables or 12 months.
MattGPT says: We do the same thing on our long form. That's going to be 9 times 12. We're expecting 108 rows per 100,
MattGPT says: 9 years. And well.
MattGPT says: thank you.
MattGPT says: Okay.
MattGPT says: So long-form data is then hard to work with. If you just kind of look at it for this long enough
MattGPT says: pivot around
MattGPT says: so that we have but for certain aggregations and polling function, then we want to go with long data.
MattGPT says:  whatever.
MattGPT says: So then, if you think about some of the calculations here, right.
MattGPT says: you then have to think about one of my it variables. If we're doing improve by here. I have to reaching country in year, identify uniquely
MattGPT says: these pendulums.
MattGPT says: And then, you know, each country, each 100 is identified.
MattGPT says: Oh, yeah.
MattGPT says: So we want to know
MattGPT says: people by gear
MattGPT says: from the long form. What we have to do first
MattGPT says: group all of the road together.
MattGPT says: The web we usually have one road per year.
MattGPT says: Now, we're going to have 12 verbs. So here. So we have to group by the 4 variables. Okay? And then what we're going to then select the reported cases
MattGPT says: is that my new problem?
MattGPT says: And then we went. Probably. Well.
MattGPT says: so this will give us this is the same as what we were doing by doing a row sum 12 column.
MattGPT says: Now we're doing it by grouping chunks of 12.
MattGPT says: And
MattGPT says: thank you completely. Yes.
MattGPT says: these countries.
MattGPT says: What's the total number.
MattGPT says: but you wanted to go. They were. I'm not really interested in the pattern over the year. I'm defying polo. We just changed that last bit.
MattGPT says: Some of that the mean? Then we could get the mean per month per year
MattGPT says: of
MattGPT says: okay. So let's see.
MattGPT says: the being
MattGPT says: okay, we'll leave the rest of that for now. But that was the main thing. I just wanted to point to the difference between wide and long.
MattGPT says: and how you go from. Why, you belong in kind of no. That's usually the way it is.
MattGPT says: I'll show you how to do specific. Have a network at that point.
MattGPT says: Okay, let's get on to number 3. Okay,
MattGPT says: And this data we've seen before. And just briefly run over this, this is the Shooting incident database from the Philadelphia police department.  and here we go.
MattGPT says: okay. So this is an older version of it.
MattGPT says: Have around, but only goes up to October.
MattGPT says: And 2 years ago. Hey? This goes to the data side. But download the latest version that we'll bring about same point.
MattGPT says: not really 21
MattGPT says: and you're a member at this time. So from the beginning of the pandemic
MattGPT says: there was a report very increased in the US. Murder rate gun violence increase.  so what I want to do.
MattGPT says: Load that notebook and then
MattGPT says: those the data end.
MattGPT says: Okay,
MattGPT says: this data is
MattGPT says: generated by the Philadelphia Police Department and Philadelphia Peace Department a little bit before is organized in districts
MattGPT says: 22 districts.
MattGPT says:  okay. So
MattGPT says: now, if you get this kernel dying thing.
MattGPT says: that's I think we talked about it before then just clean up your your kernel space. I'm going to shut down all of these networks.
MattGPT says: Wrong with that.
MattGPT says:  that's why I started.
MattGPT says: Okay. So this first date sheet is
MattGPT says: in Philadelphia
MattGPT says: all the different costs Google Covid.
MattGPT says: when we in more detail, when we go into Geo location. But the the extension of pandas that we're using here is something called geopand.
MattGPT says: So when we import geopanas and say Gbd.
MattGPT says: at Dpd, it works just like
MattGPT says: pandas, apart from it has geographical magic editor. So if you're loading in a pandas, a geoplanesave thing. then we'll have the geography geometry cost.
MattGPT says: and what's the geometry column called the Each Road.
MattGPT says: It's how is this data? Geo located. That could just be that. It's a point got along the latitude. But in this case each row represents a peace district.
MattGPT says: and then a district is drawn by a chief.
MattGPT says: That's the polygon
MattGPT says:  So the nice thing is that when no, there's a geopan data frame. When you plug, then it's just gonna throw
MattGPT says: take the geographic on, and where it's a point they will draw the point, if it's safe than ever.
MattGPT says: So that's happening.
MattGPT says: Get those we can see where the districts are. But we have all the same functions that we're now learning
MattGPT says: for a geodata frame, as we view for a regular data frame.
MattGPT says: Okay? So we've got absolutely preferences. And then we looked at the data frame of number. We've got this script and number 4. This number seems to be the one to use, and because they're trying to describe it
MattGPT says: and and look at the unique value.
MattGPT says: And they're just numbers.
MattGPT says: Okay? So those are these districts.
MattGPT says:  we don't know where those are right. And so one thing we might want to do to get a sense of that is just to label these.
MattGPT says: okay, so each of these shapes do a little bit going through on this.
MattGPT says: But we have a bullet on the center. The X and Y position thing in the middle of that point is at this time that employee attributes have wide traditions.
MattGPT says:  and then
MattGPT says: see now where the
MattGPT says: so they're all put it that way.
MattGPT says: It's 1 70 months that would be.
MattGPT says: I know where that is for that product. So
MattGPT says: I don't think one of the district doesn't have any.
MattGPT says: including roses in it. I think that's just because they don't record in that area.
MattGPT says: Okay? So that gives you a sense of what we're looking at here with our shooting data frame.
MattGPT says: Okay, let's load that in again.
MattGPT says:  you can see, we have 21 unique district. They have a variable to.
MattGPT says: There are 22 potential districts, but it only has a volume of incidents in 21 of them.  about that bit. Figure that out
MattGPT says: where they are. Now, when you.
MattGPT says: when you have a new merit variable like this
MattGPT says: and and a data frame load in.
MattGPT says: If it's you think that these believe will be loaded as strings right? And that may be the way that one. These are not numerically labeled right? If one is 2 different nominal
MattGPT says:  But what it's doing is it looks down data and says, Okay, I've seen numbers here.
MattGPT says: The one who's finding
MattGPT says: value here and a n on that
MattGPT says: where we got missing data. So some roads and we are don't have a value specified for.
MattGPT says: And so that part of the plank
MattGPT says: at
MattGPT says: so when?
MattGPT says: Right?
MattGPT says: Okay, why, bit of flooding and that big brother
MattGPT says: district?
MattGPT says: Because there are some things
MattGPT says:  But when you look at the unique ones and you see it now and there. and that was going on
MattGPT says: right
MattGPT says:  so we might want to know how he's shooting, right?
MattGPT says: Are distributed part of the difference.
MattGPT says: So really simple way to do that and
MattGPT says: is, stop your data frame. That's your variable.
MattGPT says: Remember, it looks like a number, but we're treating it as categorical. So we use value now, and we're not gonna take a mean of it back on to
MattGPT says:  and we can take our value counts
MattGPT says: usually that just comes out in as an in-depth. But value. We kind of have the 2 frames
MattGPT says:  to the end and then turns your result into the data frame. So you need the new index here. And this is your variable that we can get.
MattGPT says: Okay. So we just created a small data frame with one column where the index now is actually the district
MattGPT says: and our variables will choose count. This is the number in District 25,
MattGPT says: with the most shooting in our data set, followed by 22, followed by 24, 35. Okay, so then go back to where others.
MattGPT says: That's 25. That's 22. That's 24. If you're getting a sense of where they are.
MattGPT says: Okay.
MattGPT says: okay? And so let's turn that into a
MattGPT says:  and let's give that a name so that we can use that here. Okay.
MattGPT says: a data frame with district. And then 2 counts.
MattGPT says:  okay? And again, this is where
MattGPT says: you have this idea of voting. Okay, so, but what is a budget? And that is where you've got 2 data frames. Thanks for the grant that I want you. hey, thank you.
MattGPT says: At least one column
MattGPT says: one.
MattGPT says: And then we've got some variable 2 here
MattGPT says: don't want whatever, yeah. and then here have the
MattGPT says:  they've got all the same once that we haven't got variable breeds
MattGPT says: and it could be something else
MattGPT says: whatever that
MattGPT says: so what a verge does is it create the out of 2 tables joined on the common common road? Okay? So it's gonna say, like. and his rooms
MattGPT says: and and brought it. yeah. for them to be careful. And for the let's put those together.
MattGPT says: That's the error.
MattGPT says: So and then what we get as a result of our
MattGPT says: the new table. It's gonna have 3 polls. Gonna have our variable one.com and
MattGPT says: grab will do, and variable breathe after A, we'll say 11.2.
MattGPT says: Okay.
MattGPT says: B will be proven, but not.
MattGPT says: And with the x
MattGPT says: wow. Okay, so that's the function. Does that make sense? The concept of emerge? What is what we've got is, we've got 2 cables.
MattGPT says: what we need to. and
MattGPT says: the one right for the payout. So in this case, what have we got then? Well, we've got a common column here. which is the district qualifying. Okay? So we've got this as numbers yeah.
MattGPT says: And back in our Geo data frame.
MattGPT says:  look at it again.
MattGPT says: can see that we were using district number. That's the district number. Okay, so what we want to do is that we want to start with this data frame.
MattGPT says: This is our Geo data frame
MattGPT says: has a lot of them.
MattGPT says: So I'm repeating the same thing. But over here
MattGPT says: and see that it's got the location of the main police station in that district, and then it's got its geometry pop. This is that shape.
MattGPT says: the outline of that. And then we've also got the centroid of it that we've been using in there. But what we want to add on
MattGPT says: another column here is that calculation that we just created, aggregated by district. Say, what's the number of shooting for them in each district?
MattGPT says: Okay, so
MattGPT says: that's what we thought
MattGPT says: here, right? This is our data frame here, each district, and then a shoot count. So we want to merge together the geodatrain and the shooting. So how do we do that?
MattGPT says: Okay.
MattGPT says: this pd, so the standard and then the merge function. Okay, so the verge function takes data frame one data frame 2. So our first data frame is the district's. Pdf, one is our map.
MattGPT says: and the second one is that shoot shooting by district which is just that simple data frame where we have the district and and the shoe camp.
MattGPT says: So then we have to. We just have to let
MattGPT says: kind of, though.
MattGPT says: which variables are we going to use as the merging one or which variable and all of them have the same name. So you don't. Actually, that's why they put in this page.
MattGPT says: It's the column in the first data frame, which is underscore none. And the second thing to bring up there is actually indexed. Okay, so that's the the third and the full argument there
MattGPT says: say, merge the left hand data frame districts. Pdf, the right-hand data frame shooting by disk.
MattGPT says: And then in the left hand data frame, which column should I look to Hong Kong? Is this on as well? Now, in the right hand data frame? Where should I look? Will actually look and use index?
MattGPT says: Okay, I'm sorry if I
MattGPT says: run that budget
MattGPT says: employee.
MattGPT says: But let's have a look at what we get.
MattGPT says: Yep.
MattGPT says: okay. So you can see that we've got our first data frame on the left, everything that we had there before. But then, what we've got on the right one phone call, that one call on the same problem.
MattGPT says: the outdated right?
MattGPT says: So what we've done is we've added in a new column by learning it in from another database.
MattGPT says: This is a really common thing to do where you have 2 different data that you're like, oh, I've got some data
MattGPT says: organized by this post.
MattGPT says: Okay? But then I've got some power
MattGPT says: information about Zip code in Philadelphia or wherever you're working. I have the population in the Zip code or the number of recreation sites within that or
MattGPT says: you know, whatever it may be, that you want to bring those 2 data sets together so that we can correlate those 2 variables.
MattGPT says: What we now want to do is use that shoot count as  The way that we want to visualize these data.
MattGPT says: Okay, so
MattGPT says:  didn't say.
MattGPT says: now, with this new data frame that was disclosed.
MattGPT says: Let's do a pulse. And because it's a Geo data frame with that geometry problem, it knows when I take pulse. But what I want to do is just to draw the outline.
MattGPT says: Okay? But I wanted to do something else, too. I can also specify a problem to say, use that as the color bill.
MattGPT says: So you can see what it's doing is it's using
MattGPT says: pretty ugly. But each district
MattGPT says: how is it published by the number of shootings. In that.
MattGPT says: Okay,
MattGPT says: And so we can follow through the same steps there instead. Oh, The number of shootings. If you go back to our shooting data frame and say, actually, I only form a number of shootings. I want to know the number of fatality by digit.
MattGPT says: And so
MattGPT says:  here to where I'm going here, going back to my
MattGPT says: shooting. Not as good yet.
MattGPT says: Binary variable, whether 0 or one. If the shooting incident was fatal, then as a one. But let's first subset. Shoot the F
MattGPT says: by fatality only, and then again, this group by the district
MattGPT says: count up the number of rows that we have, and that is for day to brain. Do those things. That's again
MattGPT says: fatal. Shooting by district learns that altogether.
MattGPT says: and then we can plot this. And just so we can actually kind of see what's going on. Let's put those numbers back on top so that we can see going about the light.
MattGPT says:  where it's lighter than ever. More to
MattGPT says: drop it over.
MattGPT says: Okay,
MattGPT says: alright. Last little bit before we get to. You're working with these later
MattGPT says:  when we do boot. So far, we've primarily been using one variable.
MattGPT says: Okay? So you can group your data by, you can group your data frame by many variables you want.
MattGPT says: and you just give it a lift. Okay? Again, remember what group I is doing. Well, we have data
MattGPT says: like this, like we say, group by variable free, saying.
MattGPT says: this road and this row and this road we'll talk about that. Have 3 roads at that end.
MattGPT says: and then they throw in it for not a little but wine one.
MattGPT says: Okay, but if we had a third person the second variable in here.
MattGPT says: there's like E
MattGPT says: use. You can't.
MattGPT says: Then it would provide a combination? So you can size up your data or group your data by as many variables as you want. Now in our data, this could make sense that we want to look over here instead of what we were doing. So far with all of the data that we had from
MattGPT says: time range, and so.
MattGPT says: and see if we do that. you create a table like this.
MattGPT says: Right? So we're saying, first group by year
MattGPT says: and then group by district and then count up in each combination of those 2 variables each year, each district. How many shooting but
MattGPT says:  and
MattGPT says: but egg of this unstuck? Yeah.
MattGPT says: we just do it like that. We can see that we're getting data in long form.
MattGPT says: Okay? So you can see. yeah.
MattGPT says: And then we get the district. So each row in this format. Here is 4 for a specific year for a specific district. So there's going to be 21 per year, and then please
MattGPT says: preventing alcohol.
MattGPT says: Okay?
MattGPT says: that's the that. Maybe we we do. We might want it in long form.
MattGPT says: Want to be
MattGPT says: the one that you've edited onto the project level 0 will mean
MattGPT says: year is 0 descriptive one. Okay? So you can see. I want to have my rows be district and my columns
MattGPT says: years and right away. We know there's an error, an error ending theater 2125 mainly seem to have written values apart from one district. That's probably like one shooting that was supposed to be.
MattGPT says:  that would be.
MattGPT says: and that
MattGPT says: hold on anybody.
MattGPT says: If we change the level to one.
MattGPT says: then we would get wide data, but in the other way around.
MattGPT says: Okay. So you see the the rows?
MattGPT says: Yes, that we have represented. And then we have one
MattGPT says: columns at each of the district values. but in this case it probably makes sense. The group is the other way around. Aye.
MattGPT says: this is the the interview, and to look for patterns of their
MattGPT says: for a line chart. This is gonna work by the poll. We gave this the panel that we get.
MattGPT says: a.
MattGPT says: we would get above the cost.
MattGPT says: But
MattGPT says: a
MattGPT says: love, a little bit of stuff before you move in. Okay, so let's look at another variable in the data frame.
MattGPT says: Okay, so we're always going to find missing value in real data.
MattGPT says: Okay, so this is a option,
MattGPT says: and the way people want to test each row
MattGPT says: each row in the time problem is the value missing.
MattGPT says: and that will give you back either through a quote.
MattGPT says: and the nice thing I draw is 2, evaluating the one and both 0 that you sum those up. Then you get the number of values
MattGPT says: so that 88 roads in our data frame where the time.
MattGPT says: So
MattGPT says: where we have doing something like a mean, unless we can remember to drop the mean.
MattGPT says: I drop the missing values. Then the V will come out as missing.
MattGPT says: Okay, but on what we just want to do is drop rows where there are missing values.
MattGPT says: And so this is the growth and a function.
MattGPT says: And you can say by default, that will drop a row where any there is a missing value in any of the columns.
MattGPT says: and so.
MattGPT says: depending on your data, and how many missing values there are, you might not just want to do it straight out, drop in, but drop me more, or you want. Usually in the context of a specific query. We're working with time variable.
MattGPT says: So they broke an A, and then the subset is
MattGPT says: drop the roads where there's there are missing values in the time variable.
MattGPT says: Okay? And then select the the remaining one, the ones that don't have missing value.
MattGPT says: And then we just wanna get the
MattGPT says: the
MattGPT says: what we're getting out of that
MattGPT says: the time of day. Yeah, it's what we're trying to do here. We're sort of slicing out the time of day. I mean.
MattGPT says: because this yeah, this is an hour cool like minute. Go on
MattGPT says: second, right? I'm embarrassed.
MattGPT says: 24 h call so where the value of 50 treatment strings like our first comparative. and then you can see what time of day on those 2
MattGPT says: happening. Thank the 90'clock, 110'clock, 100'clock. 80'clock, 70'clock.
MattGPT says: one in the morning. So thank
MattGPT says:  And then we could
MattGPT says: creative all that.
MattGPT says: Okay, so this is kind of the threatened disk. Midnight one Am. 2 Am. All the way through 11 Pm.
MattGPT says: The remote.
MattGPT says: I'm most likely fine to be involved in a meeting.
MattGPT says: 11.
MattGPT says: But then that we don't want
MattGPT says: early morning.
MattGPT says: 7 in the morning, 8 to 20 that are reduced to.
MattGPT says: maybe doesn't want to look for packs like that.
MattGPT says:  okay, let's do this one. This is the
MattGPT says: answering a question, I guess, which is
MattGPT says: probably more likely to be killed. Both in the shooting, depending on the fact that right? And that makes sense, probably right?
MattGPT says: There isn't any.
MattGPT says: Okay. So
MattGPT says: What I want you to do now is we're going to split the groups. Okay? And
MattGPT says: you can take on. start off with one of these questions. Okay? And then if you get the answer to that, and you have some other questions
MattGPT says: on the data that you want from. But you can do that to hang up.
MattGPT says: Let's say
MattGPT says: this, you guys, I had before you'd be good one. We bring any good students we've briefed.
MattGPT says: and I think the entry at the back will be
MattGPT says: 4, and then
MattGPT says: thanks. And then you see that there's a question. Take on the learning. and I've given you the key columns and outline the steps. So the
MattGPT says: your group is responsible for your question.
MattGPT says: Okay? And you're gonna pull back and
MattGPT says:  there's what you did and what you got.
MattGPT says: If you find that you can do that really quickly, you can do one of the outgroup ones, or you can generate your own individual device and look at these data.
MattGPT says: Okay.
MattGPT says: so go to any
MattGPT says: alright
MattGPT says: workloads.
MattGPT says: Thanks.
MattGPT says: So
MattGPT says: we have to do.
MattGPT says: Let's do this.
MattGPT says: Why are they the first one
MattGPT says: Lysander. I'm like, are we all going
MattGPT says: personally?
MattGPT says: That doesn't happen.
MattGPT says: Shut that
MattGPT says: like Jupiter.
MattGPT says: But that's not.
MattGPT says: I'm gonna stand with your helmet
MattGPT says: registered district.
MattGPT says: But this may be
MattGPT says: as we're creating to.
MattGPT says: and it was cool.
MattGPT says: This is
MattGPT says: polls.
MattGPT says: Oh, no.
MattGPT says: sure, all because it's already September 6.
MattGPT says: If they should give one list. Let's leave the 2 brackets in the panel.
MattGPT says: Yeah.
MattGPT says: I'm trying to understand.
MattGPT says: I also did that.
MattGPT says: And then size that.
MattGPT says: Oh, wait a minute
MattGPT says: portion.
MattGPT says: Okay? Awesome
MattGPT says: shoot. We're gonna be.
MattGPT says: what are you on your
MattGPT says: yeah.
MattGPT says: we're looking for increasing
MattGPT says: function discipline. Just like, so
MattGPT says:  and then
MattGPT says: that's that's
MattGPT says: then you can use the map
MattGPT says: and then figure out, the
MattGPT says: yeah, you want to do.
MattGPT says: That's why I was thinking.
MattGPT says: yeah, I was like, why.
MattGPT says: everybody you get better.
MattGPT says: Okay.
MattGPT says: accepting us. So
MattGPT says: however, we need to get out, which
MattGPT says: maybe
MattGPT says: notes.
MattGPT says: What what do you have is mail.
MattGPT says: We also hear that.
MattGPT says: Yeah. So not to buy
MattGPT says: directly. Speaker
MattGPT says: Steve.
MattGPT says: Oh.
MattGPT says: then you can apply those to the
MattGPT says: oh, yeah. Oh, I would see, plus
MattGPT says: right. And that's kind of we interesting? I don't know whether that's nefarious or whether that's just the thing.
MattGPT says: Why would they know
MattGPT says: which is like.
MattGPT says: that's right when we can't see
MattGPT says: so? How do we know?
MattGPT says: That's again the right perfect by year?
MattGPT says: Okay, we're out of time. But we'll pick this up again at the beginning.
MattGPT says: Yes.
MattGPT says: Okay, cool. Alright.
MattGPT says: We're ready. Okay.
MattGPT says: everyone else ready to go. Right?
MattGPT says: Yeah. Okay, cool. Hang on. Let me get the
MattGPT says: we have to remember what Atlanta will just.
MattGPT says: Oh, no.
MattGPT says: we need to do. This line
MattGPT says: question was increased over the years.
MattGPT says: What do you do? I have to do it. Well, it's not what you did. And then you explain how you did it. Yeah, the first thing we looked at was,
MattGPT says: let's just look at the wrong numbers. So that's this first graph here. from.
MattGPT says: and that is showing that indeed, in 2,021, as we know shootings did spike but then, if we looked into the proportions total, abstracted by the legend here, but that's percent male percent female. Yeah,
MattGPT says: that. generally speaking, the proportions have stayed
MattGPT says: relatively flat. Yeah, there's slight change. Slight variation is a
MattGPT says: that's like female is going out slightly over time, but not very much.
MattGPT says: Okay. Cool. Now, what do you think about the this? This you've got group by year, and you got too far. So that's kind of.
MattGPT says: and I guess I'm all right, and that does compare them. The other option would be the second. Can you copy? Can you put in this one again
MattGPT says: equals. True. Let's put that in at the bottom that we've got both to look at.
MattGPT says: But can you leave that one and then copy it and then succeed.
MattGPT says: and then you're missing. I can put firm, Steve, on your show. There.
MattGPT says: that would be. Why, it's not liking them.
MattGPT says: Okay, then post that people. True. There. Okay, we'll just leave this up check so everyone can see what they did here. Right? But you, you know what we did in terms of signing from the database.
MattGPT says: Yeah. So first, we create a data frame that is conveniently named group where we grouped all the data by
MattGPT says: year and by sex. Just to. you know, have a data frame that looks at it.
MattGPT says:  I would. Yeah, I would create a new cell to to show this. But basically, it's quantifying
MattGPT says: in each year. What are the bronze bisects? Then here we added columns to that data frame that are percentage
MattGPT says:  basically having a duplicate of that data frame with, you know, percentages of
MattGPT says: percentage that's male female victims and then here are the bots. Looking at the male and female column phone numbers
MattGPT says: and then I'm looking at 8%.
MattGPT says: So a couple, yeah, if you're worried about kind of where that is, just go back up, and I'll show you on the last one. Go back up to the so before the show. adding a but plot legend, so just doing them. Well, it could be in that one. Yeah legend. And then the argument is so weird. Bb block bbox, so bbox
MattGPT says: underscore to Theo hunters go anchor.
MattGPT says: But nobody went equals, and then you give it a tuttle. So in parentheses and actually one one
MattGPT says: that pushes it outside right around that that pushes it outside the axe.
MattGPT says: But so this is like 1 one you can try like one. one and a half. I can't remember which way it goes, but it moves it around a little bit. 1 5 is going to stick it way way around that right there. So they try to try 1.5.
MattGPT says: Welcome up. I'm fine point for that. Yeah. I think it's between 0 and one right?
MattGPT says: Regardless. I think we'll learn.
MattGPT says: yeah, that's roughly paramin a minute. So that's a good way. If you got a legend and it's overlapping.
MattGPT says: just add in that.
MattGPT says: Then, okay, so what do people think? But what do you guys think about the difference between like this and this for exploring proportions.
MattGPT says: I just had a question like when
MattGPT says: more advantageous to use the staff, but also like this is another.
MattGPT says: because the spending point for the second data is like.
MattGPT says: not consistent and very difficult. Any trends.
MattGPT says: I think. This. Yeah. Well, a couple of things you can. You could stack this. You could stack this one here right? We just took count numbers, but that is exactly especially you've got lots of variance between the years that's really hard to do that. So stack, lift one.
MattGPT says:  we do the same one. Yeah, it's doing the same one just trying to look at that again.
MattGPT says: Good question, though. Yeah.
MattGPT says: So it's couple of things. See? This one like you can see, I think, what your point is here is sort of like, Wow, it's kind of weird. Because, like, suddenly, this jumps up here. And like, is this bigger than this? Yeah, it kind of looks like this.
MattGPT says: I think when you've got 2 categories, you can probably do the visual comparison.
MattGPT says: I mean what this is is basically a whole series of Piper right when we says like Pike out, don't use them apart from
MattGPT says: that idea right now.
MattGPT says: argued advantage of the budget that you can put more sections in right? And so say there were 4. Say, this was another category that has 4 values to it.
MattGPT says: Then you can stack all of those right, but that then, does become difficult. But I think in this case
MattGPT says: that's fine. I think in this case it's like.
MattGPT says: Yeah, I mean, you can probably win that. But that starts to get a little bit more difficult to interpret, maybe, but still it's probably better like this than having free bars here like it's the same on this one. Then, you see, you've got yeah, you've got.
MattGPT says: You've got 7 years and times free, right? So you've got 21 bars that you've only got to sort of interpret  But where I think it's really powerful is where you normalize.
MattGPT says: If these are proportions, then this, I think
MattGPT says: this to me is better than that. But what other people think like this is so. These are both proportions
MattGPT says: of the total number of shootings in that year. Male female, male to female.
MattGPT says: This is within the bar. How many people think
MattGPT says: this one, how many people for this one for that one number one and I number one.
MattGPT says: Okay, what about there? We're at this one for this one.
MattGPT says: Okay? And we'll kind of under budget.
MattGPT says: But I, you know, I think that's what comes back to? Why, for for those of you who prefer this one, why do you prefer this one?
MattGPT says: I just I just don't.
MattGPT says: I think this one could be able.
MattGPT says: You can look down the top. Yeah. But like, once you've been in that third thing, it becomes, yeah, virtually impossible. At that point I'd rather see like if we're looking at the change in different groups. Say, when they're stacked, they're not stack when they're dodged. It's like much easier to look at if brain was assigned than a side of the side.
MattGPT says: But the 2 like this isn't like that. But
MattGPT says: yeah, and I, you know, like you can see this, maybe highlights. The trend is kind of what I think. What you're saying is like. There is a slight bit.
MattGPT says: and the number of male victims is very slight, but it is like the best pending down.
MattGPT says: And this might be a place where you might draw 2 lines in terms of this.
MattGPT says: Okay, it's exactly because of mine. Yeah, you got lines right? And so that that one actually communicate probably better than either of these 2. If that's what you're trying to do. Right?
MattGPT says: thank you. So great. Okay, this is really good work. So one other thing I say is that you can go back up to your code right? So this birth line here.
MattGPT says: something like group by year, sex or something, you know, kind of wordy, but like
MattGPT says: name, should be readable and meaningful, or you can just say, by year and sex
MattGPT says: categorically.
MattGPT says: and then size is just a way of saying, count up the number of rows that gives us that log of that long form nested index. And so we unstack so that we go around to Y. But then pops really nicely.
MattGPT says: Another way to do that. You put in another style above it.
MattGPT says: Yeah. Or, oh, another cell. Yeah, picking on butter with those. Okay? So now, just do. Just do shoot. Just put gear in the grouping.
MattGPT says: Yeah, and then select sex wherever I hit.
MattGPT says: Okay? And then you don't value him.
MattGPT says: Sam Fishword, yeah.
MattGPT says: okay, we're on that.
MattGPT says: That will give you the same data.
MattGPT says: What's nice about these? So yeah, in that way, these are exactly saying, and you can understand that and get to that. But value counts has an argument of normalized. So to put normalized equals. True.
MattGPT says: Yep.
MattGPT says: okay. So I mean, it works. It's fine right? But then you get the proportion by your first grouping.
MattGPT says: So this is, you know, you can group by multiple things here, and then by the third, variable or whatever. But like, if you this, what this is saying is within each grouping. give me the lines.
MattGPT says: part of that. But
MattGPT says: up one top doesn't want. I think.
MattGPT says: great, great work. Alright. Good stuff. You guys, why don't you want to share your team? And when you move to?
MattGPT says: Yes, so you were doing. and
MattGPT says: what time of day we're shooting the most common in the last 3 years.
MattGPT says: And I'm thinking about this earlier right doing panic with my other class, I mean would happen to go use this data as well.
MattGPT says: You know, focusing on when we think about people, that data, what this actually means these numbers is horrific. Right? So I promise we will do more cheery data set.
MattGPT says: But yeah. how many people have been to the blue shop. Then what's going on?
MattGPT says: Okay, you should go Friday. But the reason I mentioned it is that
MattGPT says: I was rushing around. I went get one flu shot. I was on my way back into the building and going out with our running group.
MattGPT says: and I just my wife texted me and said, Hey, how you doing, and I was like just got shot right. And I mentioned just got flu shot and then what happened like with that side coming into the building, and some of the man I said this got shot as I walked into the building like I bumped into someone I hadn't seen in like 6 months.
MattGPT says: maybe even a year, if you have a baby.
MattGPT says: And then I thought she was trying to call me right, and so she was she. Then if you look at the the group. She's like, what can you clarify? What's going on?
MattGPT says: Do you need an ambulance, and then I finish this conversation. And so it went out. And then I had to pull over
MattGPT says: content. Right? Because,
MattGPT says: yeah,
MattGPT says: contacts is important. And also just like when we're dealing with data like this? Obviously, yeah, there's a real
MattGPT says: tens of thousands of of real like, trauma behind it.
MattGPT says: Okay, good. So question 2 or what time they were shooting. Okay, so yeah.
MattGPT says: what are you doing? Put it like, you can do like 6 feet. And so what? It shows eyes.
MattGPT says: I'm not sure. Why.
MattGPT says: okay? So
MattGPT says: oh, I see what you've got. Okay? So you're you're saying this is 30'clock till 60'clock. 70'clock till 100'clock. Is that right
MattGPT says: done through revising when you go back.
MattGPT says: I don't know whether it shows that each hour it's
MattGPT says: it doesn't like. Show them as being
MattGPT says: show on the bid every time
MattGPT says:  yeah, okay, so your hour is
MattGPT says: time, string 2 as in
MattGPT says:  can you can you just can you can we see what time. Ben, looks like.
MattGPT says: okay.
MattGPT says: yeah. So that would work. Let's go back on guarantee. You've got fang bins.
MattGPT says: Okay. you need to put this into your data frame. I think so. It says, shoot underscore the oh, no, or whatever you're calling them a past year. Yeah. Right? So past underscore year underscore the yeah.
MattGPT says: And then in square brackets.
MattGPT says: bins or time bins.
MattGPT says: Yeah, equal time bidding
MattGPT says: and then put bins in there as a field.
MattGPT says: But yeah.
MattGPT says: what were you were you doing that? You were saying our. oh, yeah, no, you just like you don't want anything there. You just want to group the data frame by that. So say, dog group by. So delete that popping
MattGPT says: okay? And then say, bins in here.
MattGPT says: Okay.
MattGPT says: I just get rid of the pop, for now let's see
MattGPT says: one room
MattGPT says: for them to vote against.
MattGPT says: Can I sit down?
MattGPT says: Allocate a rate of okay?
MattGPT says:  that's something that.
MattGPT says: Oh, yeah, you need to select something here. So what are we counting on?
MattGPT says: You just replace that with size instead of value counts.
MattGPT says: I don't forget a point. Yeah, there we go.
MattGPT says: Okay, so this is showing across across the past 3 years. That's the number of shootings in the utility event.
MattGPT says: Okay? So the value counts. You need to have a field do value counts on. Otherwise you value value counts on all of the the fields there. And that's why we're giving you that in.
MattGPT says: Okay, so that's what you were looking for. Right?
MattGPT says: Great. Okay, so what's your summary? How do you interpret this chart?
MattGPT says: So I guess, like, there's that shootings to. Okay, yeah, both.
MattGPT says: It's it's like.
MattGPT says: yeah. So 80'clock to 110'clock. where it would be the motions
MattGPT says: followed by 4 Hill
MattGPT says: 7
MattGPT says: right? And then be that where you gonna keep it. Yeah.
MattGPT says: Good. Okay. Is there anything else there?
MattGPT says: No? Alright good work. Yeah. Good work.
MattGPT says: You guys have one. You want to say
MattGPT says: you want to jump off that.
MattGPT says: Let me disconnect you, madam.
MattGPT says: this is okay, is it geographical distribution?
MattGPT says: So so if you wanna take, if you want to jump off this bike.
MattGPT says: put it down that tab that you had that on almost some of the
MattGPT says: better. Okay.
MattGPT says: appreciate. So you want to join. Look for 8 on the Wi-fi, yeah, and then go to display
MattGPT says: conflict where it needs.
MattGPT says: so that you you look. You may have that in your menu bar, otherwise setting display.
MattGPT says: And then it's like screen mirroring or screen sharing on the link.
MattGPT says: and then the number just pop up there.
MattGPT says: I'm sorry I don't know. I'm going to copy and paste it. Let's see.
MattGPT says: Connect to the cards.
MattGPT says: Okay. Now go back to here.
MattGPT says: Why, I was just laying the way.
MattGPT says: Okay? Great. Okay?
MattGPT says: Alright. So
MattGPT says: yeah. Why don't you? Alright. So we created 2 data frames one labeling the or one separating out mail shootings and email shooting yet.
MattGPT says: but we found the counts of those grouping them by district. So we have a list of
MattGPT says: number of shootings per sex per district.
MattGPT says: Then we applied that with the merge function to the the district. So yeah, like, 1. One thing here is that you don't really need to group by set right because they're always gonna be. What cause you? Already, this is fine. You create 2 soft data banks. Yeah. So you could just probably jump straight to these.
MattGPT says: Yeah, I mean, get a group by table and then separate them out. But wait for this, probably for this.
MattGPT says: If you're gonna go on and then sort of merge them in this way. Just do that, and then you just through by this, or you could just be value the same as this would be like.
MattGPT says: whoever's dead.
MattGPT says: We're getting errors with that, for some reason we're like talking about it.
MattGPT says: And then
MattGPT says: I think that will be just. You want me to go to the index, and that says I'm not saying the merge is. and error
MattGPT says: that's because you're merging. Oh, right on in there.
MattGPT says: I don't know, anyway.
MattGPT says: Well, we're gonna have a look at that. But yeah, okay, fine. So then you've got 2. You've got basically 2 data frame. Yeah, which is accounts.
MattGPT says: we're in that by district. Okay, even part of those. Yeah, by the count of
MattGPT says: shootings per sex per district. Yeah. And it's that of these 2 graphs which are very similar.
MattGPT says: If you look at the, the, the key on the side, the values of where they differ. So the distribution of shootings is pretty similar across sexes, like
MattGPT says: lots and lots of shootings in North Philly less so Center City.
MattGPT says: West, some West Philly, but like males, the key, I think Max, is out of a thousand. Yeah, for females. It's 120. So that's really discrepancy lives. Yeah, for you. Probably this is good. The thing is by the geographical distribution. So this is sort of like, yeah.
MattGPT says: great. Okay. So one other thing. Let's go back up.  Just think that
MattGPT says: copy these, probably your 2 box lines here like these 6 blanks and put them in another cell below and just show you something else.
MattGPT says: Show everyone coming out. Okay, let's copy that. Okay, and then put it, create a new cell below.
MattGPT says: Okay? And then at the top of that. Sorry
MattGPT says: at the top of that. But up here above the mail
MattGPT says: the rest of the topic.
MattGPT says: Yeah, it's it's we'll get on that. Yeah, we got it. Okay. The person's black made it great. All right. Scroll up and then type in big fide.
MattGPT says:  comma
MattGPT says: sorry. It's yeah. It's fine
MattGPT says: figure from an ax the axis equals.
MattGPT says: and then it's going to repeat, POP. So
MattGPT says: yeah, sorry about that. I might sometimes TLT. Dot
MattGPT says: subplots.
MattGPT says: Okay.
MattGPT says: okay? And then in premises, what Subplot does is if you want to have
MattGPT says: more than 1, one, more than one plot in the same bigger. Okay? So the way that that works is that you have a figure and that has a size.
MattGPT says: But you can arrange it. But if you have a
MattGPT says: one row, 2, but or a 2 by 2, or you know a 5 Byte. 4, or whatever, however many costs that you have. Okay, so this is what this subflop function does, and then you give it. The number of rows and the number of columns to put one comma, 2.
MattGPT says: Okay in that.
MattGPT says: So what you're getting back here is just the overall figure. And then access is actually going to be a list. Well, of the
MattGPT says: the axes inside the clock. Okay, so axis, where bracket 0 is going to be the first
MattGPT says: practice and access where bracket one is going to be the second. Okay? So now, what you want to do is
MattGPT says: add to your plot statement. Yeah. After the cmat comma a X equals
MattGPT says: and then ax web bracket 0. So you're going to plot the male one on the left.
MattGPT says: Yeah, and the lay, and then the same thing but one
MattGPT says: 0 almost there and then great. And then just drop this show right here. Didn't Dundee that show right now
MattGPT says: there's a basket on him.
MattGPT says: Okay, let's try that people again. Probably gonna come out really well.
MattGPT says: Oh, it's still loading.
MattGPT says: Yes, I moved. Yeah.
MattGPT says: okay, so everyone can see this. It's like, this is a really common thing that you'll see when you look at examples
MattGPT says: that you see it, or whatever using this subplot thing. And so say, you want to do multiple like 50'clock or 100'clock, and you say, like, how many?
MattGPT says: I wanna have 3 3
MattGPT says: columns per row. Or say, you've got a 9 box right? You can do a 3 by 3. Then you just do 3 from a free, and then you get back this this list when you
MattGPT says: I think this book there we go.
MattGPT says: They're kind of over. Maybe
MattGPT says: axis should be one, maybe. Oh, yeah, yeah, sorry. That should be one. Yeah.
MattGPT says: Okay. So it's kind of worse than that. What you can do. It's up here, put in a comma big sign.
MattGPT says: No, but inside the term seasons
MattGPT says: right here. So one comma 2 comma the whole arguments for soft problem. Right?
MattGPT says: Big size, equaled.
MattGPT says: sorry, big size equals. And then in parentheses we want this to be wider than long. So that's aspect ratio. So do something like parentheses. I don't know. 10 comma, 4 for both parentheses.
MattGPT says: The 10 comma 4 should be in front right. That works. No.
MattGPT says: it's gonna come on 1 s. I don't want to type a draw. My title also dropped. Yeah, there you go. So this is just always wanted to show you how you can get side by side for this kind of thing.
MattGPT says: II don't know arguably like
MattGPT says: visual vertical arrangement might be just as good. But here, like we've got that eyeball thing like that one that we were looking at the other week, right? And you know, say, like, here's here, it's here the overall pattern.
MattGPT says: But the thing right? You know why the title drop out of the mail shootings graph. And lesh.
MattGPT says: okay, yeah. I know why. That is so when
MattGPT says: so, this gets a little bit complicated when you have multiple axes.
MattGPT says: the plot title is actually the overruled plot. So you need to change this
MattGPT says: to the current access, right? So it gets a bit ugly. You're gonna say, as web brackets 0 instead of plt right there. Oh.
MattGPT says: and then, yeah, sorry. No, no, no, you're gonna do it over here. Sorry the actual. It's gonna be a X 0 for that saying, address that active. And then you pull the
MattGPT says: that.
MattGPT says: Okay? So that's right, and then don't step on the board by holding.
MattGPT says: Got it.
MattGPT says: And then actually, you want to do the same down here, but but active. 0 active. One.
MattGPT says: 3.
MattGPT says: It was
MattGPT says: there we go. Okay, great. And then the last thing is, you do put a Plp show at the bottom, and then you'll get rid of this output
MattGPT says: then.
MattGPT says: So I don't know, you know, wherever that textbook
MattGPT says: I mean. That's one way to go get it
MattGPT says: the other way would be numerically right. So take your value now.
MattGPT says: property dining portion. So you're saying, like, out of all of the female shootings, what proportion occurs in district A, BE line and do the same for mail. Merge those 2 together.
MattGPT says: and then
MattGPT says: people that can be part of them would add, like district labels. In this context, you can add the district labels. Yeah. So that would be like, I think that's in the main number. Or if I'm right here.
MattGPT says: Okay, yeah, you have to do that thing that we did before with centroids, and then
MattGPT says: which I think already in there and then add in, you'd have to say, access square bracket, 0 dot
MattGPT says: text. I think it is the right to text for that. So whatever look, look back to the one where I put them on. There.
MattGPT says: Great. Okay, that's good. You're right.
MattGPT says: Yep. can you? Yeah. If you just jump off by
MattGPT says:  I think it's yeah. You want. You want to stop sharing on the display. And then, oh, you need to do that. Yeah, but it probably will. It will probably go a lot. Can you join?
MattGPT says: I'm out here.
MattGPT says: Okay? So you approve. As the number of shootings of an officer involving people decreased over the last 7 years originally are their rates of disparities. When an officer involved. Okay, when do you find?
MattGPT says: So
MattGPT says: well, initially, we answered this
MattGPT says: different shooting payments? If I were.
MattGPT says: And then this is just inspecting data.
MattGPT says: And then first step is sort of it by second order, years, yeah. And then you found
MattGPT says: the.
MattGPT says: And then we created bins for each phrase, because one of the companies, the proportion
MattGPT says: of
MattGPT says: also involves shootings for different places. So
MattGPT says: do so. We start off by filtering roads with
MattGPT says: policies involved. And we created a new data frame for that. Yeah, both. Right? Yeah. And then you can just like we like filtered by each year, and then found
MattGPT says: how many cases would the doctor involve there would for a year?
MattGPT says: Yeah, and then so like the next step was to find.
MattGPT says: So we calculated proportions by race. Yeah, but there actually wasn't race
MattGPT says: marked. I don't know. Officer involved data frame so
MattGPT says: so like, you can see when we print proportion card rates, it's just an empty list. Yeah, there was no race. Try.
MattGPT says: So yeah, that could be like. maybe that's like. maybe that's only something to do. Yeah, that's some of the other ones that. or if it.
MattGPT says: if there is not an officer involved, and they do chart right? So there's if there is an officer involved in the bill right? Category. Looks like it should be a couple of days that Webinar, involved or not is a category.
MattGPT says: But when an alpha is involved.
MattGPT says: but nowhere and gender or this is rate
MattGPT says: I could look. I'm not sure you can look at that. But that's like. And I think gender actually is there where I remember? Yeah, gender is there but race and ethnicity, like there's 2 categories that we've got like black, white.
MattGPT says:  etcetera. And then we also got what non latino latino, I think, is the way that they put it in that.
MattGPT says: It's not bad, right?
MattGPT says: So like to me. That's kind of occurred, because otherwise it's dramatic, because we know, after involved shooting right? Right? And so are they reporting that data when they release these data because they don't want people to be like, look
MattGPT says: 99% or a hundred percent of the ultimate goal tuning. and both institutions are demographics.
MattGPT says: I'm
MattGPT says: could be that right, but we should look at every all way briefing and then jump good. We don't know
MattGPT says: but it could well be that like, for some reason, there's 2 separate databases.
MattGPT says: So what happens somehow is like a shooting happens.
MattGPT says: Dms plus things get cool to the site. They go to the site, probably an ipad these days like look like the form and fill in.
MattGPT says: And this is something I have to think about when we have open data ideas.
MattGPT says: who creates these data and who maintains data. And it's the Philadelphia police department, right? And they do a good job of keeping them up today. Right? It's not like some of the data set is fine here, where it currently goes up to 2018, where they could run, and then it stops.
MattGPT says: These are updated right? And you can download it today. And it's different when you know if you download it on Tuesday, etc.
MattGPT says:  But you know.
MattGPT says: when people are getting responded to in this kind of situation, that that probably really right and maybe even pass it. So I'm not sure that they're asking them. Having you identify what race you identify. Do you identify it? You know. Maybe right. But in often those are the check boxes, and the officer is something like that.
MattGPT says: Same with same with Jet. I would say. Right age. I guess they are age, but they find out age somehow. Ems, is that, too? So those details might go down to that brood pipe in that. But you just want to think about where, how these data generated? Right? And what does that mean?
MattGPT says: And that page? And so it's possible that when an officer is involved that gets recorded in a system, it gets recorded in a different way or a different form.
MattGPT says: And those 2 databases then get merged together. and they don't record alright. I mean? That's
MattGPT says: maybe I don't know.
MattGPT says: Think about that. Whenever you think about data, who maintains who generates these data
MattGPT says: and how they maintain as we talk about before, when you have categories like that.
MattGPT says: And our demographic like race, there is techniques and stuff that goes and wrong with them. Cause we're like, you know, data known data, science gets something like, Oh, this is really interesting. Let me carve up my data this way in that way, and let me plot it this way in that way?
MattGPT says: Well, like, what does that? You know? What does that get us? We gotta be careful. So.
MattGPT says: but it is, hey, yeah, that's already
MattGPT says:  your approach works here right?
MattGPT says: You made it more complicated than it needs to be. But you can actually just do a group guide
MattGPT says: on. Those do a value count right? So good thing whenever like. See what you're doing.
MattGPT says: Whenever you find yourself using a full loop in pandas, and you probably want to say like, it's a easy way to do it. You need to build yourself right? So I can show you kind of how that works out. But keep going. So then you
MattGPT says: oh, you're just looking at the increase of it. Great?
MattGPT says: Okay? Yeah. So it's a technical sense of audit
MattGPT says: by like cases with officers involved over the years with a proportion of race.
MattGPT says: Obviously, you can find that. So we just did. And
MattGPT says: okay.
MattGPT says: so that's interesting couple of things here like
MattGPT says: always when you, the line parts by default will come out with range. So the smallest volume
MattGPT says: value in your data set for the largest. But the way far part of the way seem to have the port of 0 which is guidance there like you should have that.
MattGPT says: But sometimes you may want to see what happens if you put that in. So if you add in the here PLT dot y lim.
MattGPT says: hey? Which is your wide minutes.
MattGPT says: and then a couple and then say, 0 to 25 or something like that.
MattGPT says: Yeah, what you'll see is that? Well, then. put in their appointments.
MattGPT says:  and it's worth plotting it both ways, right? Because obviously, when you've got just the range of values. that's the 0 point there. So
MattGPT says: you know. So you know, I'd like to this, because this always this always keeps the 0 where it is, which is where you want officer involved shooting to be.
MattGPT says: and it never gets down to there right? But there's something kind of interesting in this pattern here.
MattGPT says: and then you'd want to compare it to the number of shootings, and I think we see this trend going up at 2020. But it continues in 2021. The number of involved students put down a little bit there. But
MattGPT says: yeah, great good work
MattGPT says: that you wanna help up. You guys ready? Yeah.
MattGPT says: it's really good to see
MattGPT says:  So we're just quick to go through this. We're gonna stop at 4 50 today. So there's a couple of notebooks I want you to look about. Look at which, if you want, you can grab while this people are getting set up.
MattGPT says: 11.
MattGPT says: I did not elect, but I'm heavily on that deal. are you smarter?
MattGPT says: Well.
MattGPT says: yeah.
MattGPT says: we've actually, okay, that is a lot of
MattGPT says: so you can grab that.
MattGPT says: Yeah, yeah. So it's fine. So you're on Afd, cop 8. So
MattGPT says: yeah, so you need to go into, you need to go onto a Sd card 8 on Wi-fi
MattGPT says: know that one. That's it.
MattGPT says: They get 6, 3, 5, 4, yeah.
MattGPT says:  initially. Okay. Great.
MattGPT says: So you demographically.
MattGPT says: I wrote at some point. And people. And I was compared to
MattGPT says: between the trends of the parties and stuff. Okay, would you try to do it?
MattGPT says: So we've got the data and split up by babies.
MattGPT says: We try.
MattGPT says: And then we filter for you trees.
MattGPT says: Okay? And then here's a good example for.
MattGPT says: But we weren't. We were having trouble filtering right here. So give me like 2015 through 2,021 wasn't really working.
MattGPT says: but also in the data piece, and like for certain racial groups like Asian, black and white people. There were was data groups here versus like American, Indian or other, those missing for native groups.
MattGPT says: Yeah. And then here's some box sharing that you need
MattGPT says: broadcast. Okay? And here we're able to show up, maybe 17 to 2020.
MattGPT says: Okay.
MattGPT says: okay.
MattGPT says: yeah. Okay. So it's consistency, right? Right?
MattGPT says: And then here's our graph with like showing across all the years.
MattGPT says: This is a good prop.
MattGPT says: So I get for all the races.
MattGPT says: yeah, it's worth it.
MattGPT says: Okay, so there, you can.
MattGPT says: Yeah. And the for these other 3 categories, it's so small you can just about, yeah. This is an example, where Wi-fi doesn't work, and at that point fine
MattGPT says: for the same reason. Right? 2 categories.
MattGPT says: but
MattGPT says: great. All right. People hold on. Bro
MattGPT says: hopefully, you feel kind of
MattGPT says: proud of themselves for doing that. I'm very proud of all of you for being able to do that. But this question, working through the data, right? So
MattGPT says: aye.
MattGPT says: good work. So if you want to grab
MattGPT says: 4 to 11. Okay. So on Tuesday we had 10.
MattGPT says: And so there's 2 things I want to look, actually start with 11.  okay.
MattGPT says: so alright. So I told you the story about Sharp.
MattGPT says: I also after, like, I talked to Victoria and we didn't explain it. And she was like, Okay, I get it, you know. I was like, wonder what chat. So I just said, I just got shot. It's like I didn't mind, and this 3.5 I'm really sorry to hear that you can show, but I can't fire me. Help
MattGPT says: that you need simple call 9 1.
MattGPT says: This one
MattGPT says: I meant, I just got this shop. Okay, so you put this shop. Then it can't be getting shot sometimes, and then it gets it's like, Oh, let me worry. Fine!
MattGPT says: You didn't allow me? Well, you know. Okay. it sounds like you received a vaccine or a shock for a medical purpose. Thank you. So that's context and language is important.
MattGPT says: So we're going to talk more about it. And I've got reading to you, but
MattGPT says: thinking about like, giving context to numbers and to data, right? And so we talk about what is, you know, we're looking into shooting things kind of get caught up in the category thing by this way, and each row there is somebody getting shot
MattGPT says: right. And we knew we know that it's somewhere around
MattGPT says: under 20%. But around 20% like a one in 5 is fatal. Okay, so that's horrific.
MattGPT says: But sometimes we get kind of you know, we're dealing with the data to kind of lose sight of that. But also there are different ways. And we're we're really familiar with this over the last 3, 4 years of covid to see this comes from the
MattGPT says: the Cdc dashboard right? And so the number of deaths.
MattGPT says:  this is kind of giving context. So the raw number here, right as of September 20. Third
MattGPT says: is 1.1 million. Right.
MattGPT says: hey? I remember when I thought 2 years ago, and people remember this. But around this time, 2 years ago, 600,000,
MattGPT says: and in Washington after that time I think it was
MattGPT says: one of the pop. They could have that number of flags right? So 600,000 black. And that was a way of trying to make data real by saying, and and was a 2.4 mile loop to go all the way around that.
MattGPT says: Okay, so think about an hour right? Depending on how fast you run.
MattGPT says: Have to take the frame in. It takes 30 min to do that right? So it's almost like a 5 K or
MattGPT says: that. So representing the data of this. This number was Clark a bit at that point, right? It was 600,000 assignment. But when we see big numbers like this like team kind of thing. Well, wow.
MattGPT says:  that's a lot right? 1 million is a lot. But
MattGPT says: after a while it's easy to get sensitized. When we we're so used to seeing this number being over a million, or just looking at this number. So you can think of it in different ways. Right? Somebody look up the proportion of the number of people in the
MattGPT says: in the Us. And try and calculate the proportion. Okay?
MattGPT says: And so you're going to do one of these 2 things
MattGPT says: find quickly. Google report the number of people in the Us. And workout portions. From this number. And then the other thing to do would be, look at the US. Population by State.
MattGPT says: and find the State which has the closest population to this number. They would just put it through that number.
MattGPT says: Oh, and just as a
MattGPT says: a piece of python syntax. I don't think I've shown you before when you've got large numbers like this that have commas in.
MattGPT says: But it's by some syntax is to use an underscore where you have homes
MattGPT says: right? Which maybe looks a bit ugly. But it's kind of useful for proofing right.
MattGPT says: So if you put that number in, it, says, Okay, I recognize that as an integer object.
MattGPT says: So you come across a number and you're like.
MattGPT says: I use that
MattGPT says: is, anyone found the the US. Population has done the calculation
MattGPT says: one
MattGPT says: at the
MattGPT says: 23.3 4%. Okay.
MattGPT says: that about right? Right? It just said, What does that say, yeah. But you get 331, 332 million. Yeah, yeah.
MattGPT says: Okay. So comparing that number to 1.1 4 million compared to what was it? Point 3 2.3 4.3 4%.
MattGPT says: And so in some ways, like whenever you've got a number which is smaller than one that's hard for people to grasp. Right? We're used to thinking in units of one
MattGPT says:  so that. But still, if we could be so used to this 1 million number that we could
MattGPT says: throw out the other one just to get people kind of shocked. Do you remember? Sort of you know, if you know how like we respond to it? Once we once we become certain types of something, then we cannot go notice it
MattGPT says: or attempt not to think it in a bank. So changing the basis numbers normalizing them in different ways, can be a really effective way to communicate it. So anyone look at state populations. Yeah, what do you find? One pound?
MattGPT says: Okay. So the whole State of Montana got wiped out in the last 3 years.
MattGPT says: Okay, you don't even need to say what the numbers are. I think most of us who know the Us. Would have sent what? 100, right? I mean, I've never been anyone having one hand.
MattGPT says: I don't bid.
MattGPT says: Okay.
MattGPT says: it's beautiful. Yeah, it's a beautiful kind of place, right? But like the the people that right? It's not like.
MattGPT says: It's not like. There's a lot of people. When you went to Montana, when you went to Montana you probably saw a bunch of people all those people about.
MattGPT says: So that's another way to think about kind of community numbers find a different way of taking a big number like that and giving it to the person. And then you don't even need to use the number of books.
MattGPT says: People sort of say that. Okay, so here's another one. and this one
MattGPT says:  this one is deeply disturbing me, and they can say my classes when I look out of my classes
MattGPT says: for 60 plus percent female. Looking out there more
MattGPT says: and one in 4 females with very sexual violence during their college career like
MattGPT says: And so I, how many people have heard that figure before? Yeah, right? And it's
MattGPT says: we thought it was 1, 3.
MattGPT says: Could it also be one in 3? Yeah, you know, I heard one in 5, one in 3. I think. You know it's it's certainly a shocking proportion, right? But in the same time, like there's something about like one in when you hear it, a lot, even one in 4 is like, Oh, okay, you know. Not meaning to downplay that. But in that form it's still very powerful and shocking in that form, and I just would look around.
MattGPT says: And then you can just sort of. But again, you can think about different putting that number into a different. So you look up on college enrollment
MattGPT says: in 2021 like it's usually 60 plus attendance at a college entrance every year, enrollees every year. Female to 40. Something male, I think. 58.
MattGPT says:  so that's 8.9 million students.
MattGPT says: Okay, so what's the number
MattGPT says: in 2021. So they're now halfway through their college, like they go to one and 4
MattGPT says: 1 point
MattGPT says: alright.
MattGPT says: So yeah, if you found out what that was right. But I think it's 8.9 million, right? We're talking about 2.3 million
MattGPT says: of the females who enroll in college in 2021.
MattGPT says: Yeah, but you're right. If you have it, then you're exactly there. If you if you pass an equal number. Yeah, yeah, I guess. Yeah, that's a good way to put it to right by now already 1.4 million.
MattGPT says: which is shocking. Right?
MattGPT says: pretty ready fucking. And so we've we've got to finish very quickly, because
MattGPT says: in this room there is this book here. If I just want to show you this and invite you to this, if you haven't heard about it already.
MattGPT says: Okay. The new Dean of Annabelle.
MattGPT says: Sarah Renee Wyther.
MattGPT says: with a co-author, has a new book out.
MattGPT says: which is the me to effect what happened from me. But the women. Okay,
MattGPT says: And there is also another book that's just come out, though, by the Gilmore
MattGPT says: But it's going to be taught right after our farthest heat and a reception. And you're welcome to that.
MattGPT says: You're all interested. Obviously, the topic is really important, but also disturbing. So I think
MattGPT says: exactly
MattGPT says: okay. And so we'll come back to this when we pick things up.
MattGPT says: But in photos 4
MattGPT says: sorry, important, and
MattGPT says: no before I gave you and not look it.
MattGPT says: There was some plotting options. Okay? And so we see, map, what level up. And we've been using that. And that's the base sort of starting point for most
MattGPT says: type and body.
MattGPT says: Okay, so you can look through this notebook. This is the World Health Organization, Gdp
MattGPT says: and income. Okay, so this just shows doing some data. But
MattGPT says: but then, when we get the pudding
MattGPT says: our default is to use map plot. Okay, so this is what we've been doing. All of you were doing that in in this right? And so
MattGPT says: this is an extension. This is an area chart. It's an extension of the stack pop up. So it has the same issues that we book about ready. But this is the time series. So mostly see what's going on. Put on the bottom one, and then next one up and then expires as you go up.
MattGPT says: Okay, but I just kind of show you this so you can see how you can actually do that with these data. you could do it
MattGPT says: as a proportion. So the same calculation that we just did. But sure over time.
MattGPT says: Maybe this, you know again, this human, maybe too many categories that are really interpreted.
MattGPT says: But you can see that the other option would be to model pups or to do lines.
MattGPT says:  then the other library that I want to call attention to is one called Seaborn that you've used in the assignment.
MattGPT says: This is
MattGPT says: built on top of Mapbot. There. But it's more
MattGPT says: geared towards statistical kind of science that I had a slight hidden syntax
MattGPT says: where you always give a data frame and the data argument. And here I'm upsetting it, which is my tiny. And then you specify what's going to be on the X-axis.
MattGPT says: and so I think I only had one which the x-axis. But then you specify a secondary grouping, which is like cube, the color of it. So here we're overlaying a whole bunch of grants to see that.
MattGPT says: Take a look at Seabourne
MattGPT says: high data, and you can see there's a gallery and see that there's a whole question
MattGPT says: stuff that you can do with it. It has regression block built in.
MattGPT says: You can do a rich plot like this.
MattGPT says: key maps for correlation of table ward.
MattGPT says: etc., right? So it's built on top of map up there. But it's got some really nice styling in it. You can change the style
MattGPT says: and so on. Okay, and you can do things like this, which is the other often overlaying, is actually to do 5 by 5. And can we talk about? And I slide to show you how you can use subplots? And you say, active, one active, 2 after 3, or we have 3 by 3 set of rows.
MattGPT says: So this makes it much much easier
MattGPT says: because you can use this notion of facets. Okay, and so this is similar. If you've done. Gdb. where you have grids, a classic grid where you can specify the number of rows and the number of columns, or you can have a plastic wrap, which is just sort of like, just keep going. And then when you get to free, they're out of ground.
MattGPT says: Okay? And so it has all of those things in it as well.
MattGPT says: What they expressed is. I mentioned this briefly. This one works just like the Matlab plot in lots of ways quite different syntax. But again, you specify a data frame.
MattGPT says: and then an X and y, and then a variable could be a color here.
MattGPT says: and the advantage of this for explore occurring something is that you get a toolkit over there. So you get some and interactivity right? And it also.
MattGPT says: we'll then
MattGPT says: and stuff like that. Okay.  And then in today's holder.
MattGPT says: If you look at the second notebook up here, it's more following.
MattGPT says: Okay, this just shows you same little data set. This is the measles data set, which, remember, was
MattGPT says: wide and long.
MattGPT says: so let me just run all these like, see what's going on
MattGPT says: the
MattGPT says: okay. So this is our.
MattGPT says: I hate it. This is our back pot shop by region.
MattGPT says: And we're moving these instances of measles. The total instances of measles in that year
MattGPT says: across all regions of the world. And then we've written down by region.
MattGPT says: Okay, we can then go into potato express. And
MattGPT says: but, for example, again, have a look at the documentation here. This just shows you the bar charts, but if you look on  there's a gallery for this.
MattGPT says: You can see that it colors.
MattGPT says:  yeah, that covers a whole bunch of times. things that you can do, how many things that you always get is interactivity.
MattGPT says: It will let you do pipe off.
MattGPT says:  But the syntax is nice and theme right? So Px line. And then the data frame, and then the X and the Y variable from that.
MattGPT says: and you'll get an in bracket versions of cost.
MattGPT says: Okay? And so this just shows you how to do the same stack of like, either from long informed data.
MattGPT says: Which is this one? Okay, which looks like the seaborn stuff where you have what's on the X variable, active, or for the Y active, and then color is the same as huge. So on the Seaborn you see a weight of groups
MattGPT says: that, and it's that, Daisy.
MattGPT says: you can do it. This is the Y form. So we've got year per row, and then we've got a column for each region.
MattGPT says: You can do the same thing you just have to specify. X is the year, and then y, you give it a list below
MattGPT says: of regions, and it will slide up. Okay. how do people use data wrapper?
MattGPT says: Okay? A few people do. Okay, we'll come back to this. But what data wrapper is is,
MattGPT says: it's a online
MattGPT says: service where you can start up with a free account, and you can get along for the way. And it produces really nice interactive charts that you can embed within
MattGPT says: a notebook where you can get it within a web page or or post, and you just have to sort of upload your data. And that's quite a chart. That's how you do it. If you're working
MattGPT says: within
MattGPT says: within data, wrap up right?
MattGPT says: If you want to.
MattGPT says: set up an account. Okay? So you can sign up for free. Then you have to create an access token, and then you'll put that in here.
MattGPT says: But then what this allows you to do is to using the data, wrap up pipeline module.
MattGPT says: You can access your account. and you can set up a chart here. Okay, so this is how we set up a couple of things.
MattGPT says: They'd rather create a chart. We're gonna have a title, one called Meetbook by global region. What type chart type, is it? You'd have to look on the data of rapid documentation. But basically, we're doing that far out here.
MattGPT says: And then all you have to do is just give it your wide form data by photos.
MattGPT says: Rose's ex and the columns as the Y groupings.
MattGPT says: Then you'll get
MattGPT says: the data wrapper.
MattGPT says: The
MattGPT says: that was crazy but
MattGPT says: awesome.
MattGPT says: We don't.
MattGPT says: Yeah, here we go. Okay. So here's that, what that then looks like up on data wrapper.
MattGPT says: Okay? So if you've used this before. You know that you can go onto the website and you can upload data, the left, the chart type already refining and annotate it. But this comes straight now of working in
MattGPT says: panel and the department.
MattGPT says: Okay, thanks everyone. We're out of time.
MattGPT says: but just quickly, I will post the assignment for this week because it hasn't come out yet. You're gonna get around until next week. Do it fine.
MattGPT says: Okay. Hi, everyone.
MattGPT says: hope everyone's doing okay?
MattGPT says: So what I wanted to do today was to give
MattGPT says: as a chance, just to sort of review what we've covered so far, but also
MattGPT says:  and can everyone hear me? Okay.
MattGPT says: okay, everyone can hear me. Okay.
MattGPT says:  is to great. Thank you. Just to review
MattGPT says: where we're up to, and particularly
MattGPT says: see if there's questions related to pandas and the and sort of analysis pipeline that we've been working on. Okay. So
MattGPT says:  and also just wanted to make sure that people had a chance to ask any questions or use this time some of this time to work on the assignments which are due. So just to remind you,
MattGPT says: assignment 4 is due
MattGPT says: tomorrow.
MattGPT says: and also the annotated bibliography that we looked at last week. So I wanted to just call attention to that actually is first is that you'll notice if you were. If if you grab this last week and when we talked about this in class on Friday.
MattGPT says: no, Thursday, wasn't it? a.
MattGPT says: There was a folder which is just called annotated periodography, and we went through sort of setting up Zotoro and doing all of those things. So there is an updated version now which I'd like you to use. Okay, so if you want to grab that
MattGPT says: and we'll look at that. And then also, if you haven't got assignment for yet. Grab that.
MattGPT says: Okay,
MattGPT says: But first, the annotated bibliography.
MattGPT says: So the updated version is the one that you should be using. Okay, so this is the one that we looked at last week, and that was really just kind of giving you a temp. Yeah, giving you a notebook to work with. It showed you the
MattGPT says: the sample
MattGPT says: essays in there from previous. So the same 2 notebooks are in there.
MattGPT says: So if you haven't already, and you haven't got started on looking at this. Yet the first step is to pick what you want to write about in your short paper.
MattGPT says: and so there are a whole bunch of summaries and titles from
MattGPT says: previous papers that people have written. Okay, so you take a look at that. And then this annotated biography assignment is the notebook that I've updated. Okay? And so doesn't add anything too much to what we did before, but just fills in the steps that we went through. Okay, so if you hadn't, if you if you weren't in class on
MattGPT says: Thursday, you haven't got a start on this yet. This will give you much more of a step by step of what you need to do. Okay,
MattGPT says: And so the key points here.
MattGPT says:  is to get started on your short paper
MattGPT says: and use that that previous papers, the example, short papers, notebooks, should give you an idea of the range of topics that you can pick on. You don't have to pick something within that range. You can. You know anything which is related to the use of data? But you want to have a targeted
MattGPT says: interesting question, which is interesting to you. Okay, so that can cover in the things that we covered in the readings that we've talked about generally data, privacy.
MattGPT says:  use of data, and you know, to address public issues.
MattGPT says: ethics around data, AI and data,
MattGPT says: and so on. Okay, so pick something that's exciting and interesting to you.
MattGPT says: And then.
MattGPT says: what's added to this notebook? Just the instructions of how to go about setting up a zotero collection of citation
MattGPT says: citations, and how to use the the browser plugin and then
MattGPT says: the key thing, and when you get it is to annotate each of the entries that you have. Okay? So you're looking for 8 to 10 sources.  that relate to your topic and try and make at least 3 of those be
MattGPT says: academic sources, and the way to to sort of
MattGPT says: an easy way to do that is just to look in journals and make sure that it's you know, a reputable journal. Someone's addressing the issue from an academic perspective. And so it may not be. If you've got something which is
MattGPT says: you know, very specific. You may not find that there is much written specifically on that academically. But what you can do is pick out some of the key issues there. Okay, so if it's around privacy issues or
MattGPT says: the ethics, you know, related to personal data. For instance, you will find people have written on that
MattGPT says: And so look within social science journals or look within communication.
MattGPT says: But it may be that if you go to Google Scholar or you go to the
MattGPT says: the library catalog, and you know, put in the keywords for your topic, that you will see that there has recently been some stuff written. Okay. So you want to have stuff which is relevant to what you're talking about.
MattGPT says:  And then, just to
MattGPT says: to complete and submit the assignment. What you're going to do is once you do have generated your biography
MattGPT says: in Zotero. You'll export it. I'm gonna put some screenshots in here just to make that clear. Okay, so how that works? Well, I'll show you from an example one in a sec export this as a bib text file. So when you hit the export option.
MattGPT says: You want to hit Pip text.
MattGPT says: and then that would download to your
MattGPT says: computer.
MattGPT says: rename that file. It will come out as something like bibliography. dot bib or something. I can't remember what the default is. Actually, that's Zotero
MattGPT says: does. But then rename it to my bib dot bib. Okay? And then you're going to upload that
MattGPT says: to this folder, and you'll see that there's already the example file that I've given you
MattGPT says: already has that name. So when you do upload. It will ask you to replace, you know. Do you want to replace it? Say, yes. So that you're replacing this file with your annotated bibliography file.
MattGPT says: Okay?
MattGPT says: And then just
MattGPT says: add a couple of sentences here about what your paper topic is. Just so I can get a sense of what you've chosen to write on and then, once you've uploaded your biograph file, you should be able to run this test cell.
MattGPT says: Okay? And what that will do is that will print out your
MattGPT says: annotated bibliography. Okay, so
MattGPT says: that's just
MattGPT says: the updated version of of how the assignment works. Okay.
MattGPT says:  anyone have any questions about how how to get started on this, or now have been working on it. Have any specific questions on it.
MattGPT says: You just if you can just jump in and and
MattGPT says:  okay, you can put questions into chat as well. And
MattGPT says: I will try and fill those. Okay. but I'm guessing everyone, at least try to make a start on that. But the key things just to to, you know.
MattGPT says: to review. I'm
MattGPT says: make sure that you've downloaded the updated version of the assignment. Okay, so don't use
MattGPT says: this one here that we started that you had on Thursday. Grab the updated one that's or you can use it. If you have already kind of
MattGPT says: use. Put your bibliography into the up into the original folder. You can just copy it across here.
MattGPT says: Okay.
MattGPT says:  But just to
MattGPT says: remind people. If you haven't
MattGPT says: use the tarot before.
MattGPT says: this is what it looks like in case you're gonna
MattGPT says: go to this satara.org and set up. If you don't have an account set up at an account.
MattGPT says: Okay? And then you'll
MattGPT says: nice login once you've got your account set up. Remember. password is.
MattGPT says: Think it's that? Yes, it is
MattGPT says: okay. And so you'll see that you have an overall library.
MattGPT says: And
MattGPT says: and then underneath your library, you can add collections. Okay? So what I suggest you do is add a collection that looks like something like this.
MattGPT says:  and then you can add your sources there.
MattGPT says: Okay? And so then the process is, if you've installed the
MattGPT says: collect the connector for your browser. Okay?
MattGPT says: then you will have an extension in your browser here
MattGPT says: that will. You can either find you, and looking under the extensions here, or if you find there's a terror, once you install it, it will say, add or PIN to your toolbar. That's what you want to have there. So you have a connector like that.
MattGPT says: So what that allows you to do is if you're
MattGPT says: on on a website here, and that, you know, like I said, one place to go to start with is Google scholar.
MattGPT says:  and if you're looking at
MattGPT says: privacy.
MattGPT says:  and social
MattGPT says: media data. For instance.
MattGPT says: Okay, then, depending on what your topic is, you can look through the sources and so and and try and identify some papers.
MattGPT says:  that are
MattGPT says: relevant to what you're you're wanting to really talk about. Okay, and don't pick anything that's just too general. You want to have sources which are particularly, you know, particular to your question. There.  okay. But say, say, you know your topic was along these lines, right? You were into interested in in conservation science.
MattGPT says:  Then, you can see.
MattGPT says: Take a look at this paper. You get an idea from the abstract whether it's of interest to you where it's useful for your paper.
MattGPT says: Have a scan over the whole thing right? Get a sense of what the argument is. And if you're like, yeah, this is going to be relevant. You're going to select that as one of your sources
MattGPT says: and then up here, and
MattGPT says: you can use the extension here save to Zotaro. and
MattGPT says: it should give you an option here, it will either save straight to your main.
MattGPT says: Bibliography, or if you've got, if you've got collections, then this is really convenient. You can just select the collection
MattGPT says: that you want to save it to.
MattGPT says: Okay. And then, if you just hit enter, and that will
MattGPT says: add this source over here
MattGPT says: and
MattGPT says: need to just refresh it with that.
MattGPT says: And there is
MattGPT says: the how to address data privacy concerns.
MattGPT says: Okay, so that's how you collect your sources together. It works also quite nicely for non academic sources, too. So
MattGPT says: let's say we had
MattGPT says: chat Cbt. And education.
MattGPT says: and we're looking like at news sources. And
MattGPT says: and
MattGPT says: you came across this time article right? And you think that this time article actually
MattGPT says: gives you something that's useful. for your argument that you're going to have in your paper.
MattGPT says: So this will be one of your 5
MattGPT says: to 7 academic sources. Remember aiming for like 8 to 10 as a total. Okay, so at least 8. You can do the same thing here. Okay, so
MattGPT says: any website, this plug-in will allow you to  get the citation details for this article. Okay? So if we click on that and again, say, yeah, we wanted to go to our annotated bibliography.
MattGPT says: and then come back over here.
MattGPT says: You can see here
MattGPT says: you'll get some, you know. It recognizes a web page, gives you the title and the date, and it gives you the yeah. URL,
MattGPT says: okay? And
MattGPT says: so those are our key key things. There.
MattGPT says: let's go check on here.
MattGPT says: Alright.
MattGPT says: Sorry. I'll come back to
MattGPT says: Okay, so
MattGPT says: so I'll come back to your question in a sec. So that's your
MattGPT says: that's how you collect the sources together, so you can decide whether you want to find a source, collect it and then annotate it or collect all your sources and then go through and annotate it.
MattGPT says: Okay? So the key with annotation. then, is to use the abstract
MattGPT says: field for a source. Okay? And so
MattGPT says: it may be there, or it may, and and have a lot of detail in it, especially the academic ones. I don't want you to use the abstract that comes along with the source. Okay, we're just using the abstract field for you to write your 100 to 150 word summary.
MattGPT says: Okay, and so delete whatever's there.
MattGPT says: think about what you want to say there. Okay. And so again, if you haven't done one before, the purpose of an annotated bibliography is to collect together relevant sources for your paper, and then to add.
MattGPT says: You know, to to make sure that you've read the sources and you've confirmed. Yeah, this is going to be useful to my argument, and so, you know, 100 150 words is not much. But you just summarize
MattGPT says: what the paper, or what the article
MattGPT says: does, and particularly the points that you think are relevant to what you're going to be arguing in your paper. Okay?
MattGPT says: So  with reviews.
MattGPT says: classroom
MattGPT says: usage of generative AI
MattGPT says: and
MattGPT says: the potential
MattGPT says: changes.
MattGPT says: operation
MattGPT says: and rule!
MattGPT says:  presents 3 key.
MattGPT says: Pause my click
MattGPT says: argument for
MattGPT says: use, and then you would say in it? 1, 2, 3.
MattGPT says: That would that be an example of a good summary annotation to go with your sources? Okay? So then you add that into your abstract. I mean as the abstract for each of these.
MattGPT says: you've got your 8 sources here. Okay, so we've got 6 here.
MattGPT says: okay, let's let's just grab a couple more.
MattGPT says: a
MattGPT says: kind of use strategy as a learning tool
MattGPT says: sounds good. They don't just do what I'm doing, which is grabbing things without even looking at them. But I just want to add some more in here, so that we end up with 8
MattGPT says:  most schools want kids to use. Chateav, really.
MattGPT says: okay? And so
MattGPT says: add your annotation goes there. Creative ways.
MattGPT says: That's the one that we did.
MattGPT says: add an abstract. So sometimes you'll see it looks like that where there isn't an abstract already. There, you just go down to this bottom
MattGPT says: window right here, add an abstract
MattGPT says: and
MattGPT says: okay.
MattGPT says: Okay. So once you've got your
MattGPT says: 8 to 10 sources in here. Go to highlight them all.
MattGPT says: Okay?
MattGPT says: Like. So.
MattGPT says: And then click on export
MattGPT says: just here.
MattGPT says: And you want to select the third one down, which is bib tech. a, BIBT. X.
MattGPT says: Select that
MattGPT says: and that okay, so the the it will give you a name something like export dash data.
MattGPT says:  and dot bib.
MattGPT says: Okay? And so I find that in my downloads folder I'm going to rename it
MattGPT says: as my bib dot bib.
MattGPT says: Okay? And then so
MattGPT says:  back over here and
MattGPT says: assignment holder.
MattGPT says: and I want to upload that.
MattGPT says: My name is Deb. and then you can see it says my bib dot bib already exists over. Right? Yes, override.
MattGPT says: Okay. And so what? Just to to show you again what a bip file is, is it? Gives you?
MattGPT says: And
MattGPT says: it's a text form of citation, things that gives you something that looks a little bit like a python dictionary. It's not a dictionary, but you can see that it's got this key value pairs. And so each citation
MattGPT says: has a reference, a citation reference.
MattGPT says: Okay? And so sometimes these ones that you'll grab like, which are just yeah. Urls. You know, it doesn't pick up the authors necessarily, but it gives you the it picks, picks up the title, picks up the URL
MattGPT says: and the date. So that gives you enough to to make use of that  when it's when it picks up that it's an article. Then there'll be much more information in there.
MattGPT says: Okay.
MattGPT says:  So just make sure that your version is update is is uploaded. The way to do that, my bib dot bib, and you can come back to the assignment.
MattGPT says: And then you're going to just fill out these 2 things. Describe what your paper topic is. Once you've uploaded it, you should be able to run this cell here.
MattGPT says: Okay,
MattGPT says: let's see.
MattGPT says:  and don't worry if you if you do get, you know, as long as you get this message, which is, you have 8 items in your bibliography.
MattGPT says: and
MattGPT says: then it should be good. It may give you some errors along this line that some of the sources don't have authors in there.  don't worry about that. I just want to make sure that you've uploaded it, and you'll get that message now, if you it.
MattGPT says: and it's when it does work, then it will
MattGPT says: It will give you a version of the bibliography right here. But for now I just want you to get this thing uploaded. So once you get to this stage. Make sure that you filled in this. You've uploaded that.
MattGPT says: Then you're ready to submit it.
MattGPT says: Okay, so you'll come over here and you can submit the updated one.
MattGPT says: Okay?
MattGPT says: alright. Let me just share for a sec, because it seems like my screen.
MattGPT says:  okay, yes, you can discuss topic with with me. So
MattGPT says: we'll come back to that, Olivia. But yeah, I can lease some time for that.
MattGPT says: Are there any other questions that people have about the annotated bibliography.
MattGPT says: pressing one. So let's just go back to Shannon.
MattGPT says: That's how the annotated bibliography works. Okay?
MattGPT says: you should. It should be really straightforward. And the reason I wanted you to use the taro is because it saves you a lot of time and war worrying about how to
MattGPT says: collect their right
MattGPT says: data, how to you know what style to use. We we probably end up using something like Apa. But this format that I'll show you when we actually get to how to write the paper, and we'll write it in Jupiter as Markdown.
MattGPT says: I'll show you how to use your bibliography to make citations in the actual text.
MattGPT says: and then you can output when it's you know, finished, you'll run it and save it as a Pdf.
MattGPT says: And all of the citations and the bibliography will come out formatted for you. Okay? And
MattGPT says: So then assignment 4
MattGPT says: again, it's due tomorrow. This should be really easy. If you have been following along. And what we've been working on using the
MattGPT says:  the open data, the open Philly data shootings. Database. Okay. Now, the one thing that was, I mean and apologize about this. There was a typo when it was first released, and some people.
MattGPT says: I think, downloaded it. And so if you have a version that you downloaded that has some look at this first part here and it will say, Csv version downloaded. It may have the the version from February this year that we were using in
MattGPT says: a
MattGPT says: this in folder 9 when we're in notebook 4. Okay.
MattGPT says: and that's the one that you want to refer to, because that's where we've been sort of doing this data analysis.
MattGPT says: But you notice here that we used a version that was older and only had data for this year up to
MattGPT says:  about the end of February. Okay? Whereas in the assignment you should be using a more updated version.
MattGPT says: Okay? But I noticed after I posted it,
MattGPT says: or someone actually kind of pointed this out to me and I fixed it. But what happens when you fix and re-release an assignment is that only you only get the the updated version. If you grab the assign if you fetch the assignment
MattGPT says: after that point.
MattGPT says: But you can still use the assignment
MattGPT says: notebook, but just check that, you know, if it if it has that older version, and you try and load the data. Then it will give you a you won't get past question one, basically. And this is what? Where some people were getting stuck.
MattGPT says: because if you look in the data folder and you can see that there isn't
MattGPT says: the file named
MattGPT says: open Philly. Underscore data. Underscore shootings. Underscore 2, 21, dash 23. Okay? And so if you try and load data in here.
MattGPT says: like this.
MattGPT says: then it will break. Okay, so that was just the the fix. And I think I posted that to a number of places that will get you passing plus question one. Okay, I think there's there's 8 questions in here with some sub parts. It should be so, you know, it's all all about practice and reinforcement. So you should be able to work through this very quickly. And especially if you refer back to this notebook right here.
MattGPT says: Okay,
MattGPT says: So
MattGPT says: let's think about the yeah.
MattGPT says:  So what I wanna basically do use the rest of the time for is
MattGPT says: you can.
MattGPT says: you know, if you don't have any questions. Then you can go ahead and you can leave the zoom. But if you have questions, hang around, okay, and that can be either questions on the annotated bibliography or on assignment for so if you want, if you haven't started working on assignment, for, like a good idea might be to start working on it now and then, you know.
MattGPT says: if you have questions, then
MattGPT says: I'm happy to work through those with you, and you can share your screen, etc.
MattGPT says: What we can do is sort of like the equivalent of
MattGPT says: coming down to the front of class. And you know, asking me a question, is that we can
MattGPT says: jump into a breakout room if you have a question, and then you can share your screen, and then
MattGPT says: we'll come back to the main, the main room like that.
MattGPT says: Okay,
MattGPT says: So, so, yeah, any
MattGPT says: any specific questions that people have?
MattGPT says: Yes.
MattGPT says: it doesn't take it as an integer. And I think it's because of the dash in the middle. Maybe it's that's right. Yeah. So we have string. So do you mind sharing your screen, and then we can take a look at that.
MattGPT says: You should hang. Let me just check that screen sharing is turned on for everyone. I just I had to create a new account. I need to quit and reopen 1 s.
MattGPT says: Yeah, I think that's that's that's a good topic. I think that's okay. I mean.
MattGPT says: specifically within, were you gonna sort of frame it within the context cause? I know in the strike that was sort of one of the things that some of the riders were calling
MattGPT says: for some attention to, maybe some.
MattGPT says: I'll let's just a second
MattGPT says: you may want to. Just you know you could. You could frame it generally you could. You could. If you want to discuss both of those things, then you know you can. You can definitely do that, or it may be as you get into it that you feel like the ethical
MattGPT says: sort of a
MattGPT says: part of it, and that becomes the main issue that you look at. But yeah, that's fine. And and and that is broad. Definitely, you're talking about AI models. It's about.
MattGPT says: it's broadly about data. So that's fine. Yeah.
MattGPT says: Okay, great. Okay, be. So this is your
MattGPT says:  Okay? So let's see what you've got here.
MattGPT says: yeah. Okay, good. So you
MattGPT says: added the new field, create a new date field a day, a new data frame. Yeah.
MattGPT says: sorry. And there's a typo there. And it's not a date frame. It's a data frame which is gonna be called, yeah, okay.
MattGPT says: yeah. So you can, actually, you can actually use you want to do 2 things. Remember when you were creating a
MattGPT says: when you're creating a a sub doing a subset by row, then you want to create a filter. Right? So you've got your filter there, which is good.  And yeah.
MattGPT says: okay, so that should work. What you've got there is filter. October. Now, if you apply that
MattGPT says: to shooting underscore. Df.
MattGPT says: okay, so shooting underscore df, and then square brackets.
MattGPT says: Okay? And then just run that. You'll see, I think you or you were just about to do this. Then what does it say? Hang on,
MattGPT says:  like
MattGPT says: Oh, okay, okay, yeah. I see what you're done. Do you see? You see that line that you're currently on?
MattGPT says: Yeah, if you look at the question I asked you to go back to the first data frame. so is in your first line. There.
MattGPT says: you see, you've got shooting underscore. Df, equals
MattGPT says: that you actually want that to be shooting underscore df, and not shooting pre. 2020 free
MattGPT says: to see?
MattGPT says: So that should be shooting underscore. Df, can you see in the question, it's saying, create a subset of shooting underscore? Df, yeah, there we go.
MattGPT says: So try that.
MattGPT says: And now you might have to re, you might have to jump
MattGPT says: like, let's rerun the whole notebook. So go up to the top.
MattGPT says: Well, this.
MattGPT says: yeah, you can go to run and run all cells.
MattGPT says: Because because you you'd already sort of okay. So then let's see what we get here.
MattGPT says: Hold on. Scroll up again. Sorry
MattGPT says: shooting high school date stream 0 to 7. Okay, that should give you
MattGPT says: that.
MattGPT says:  okay, shouldn't be October.
MattGPT says: Okay?
MattGPT says: Oh, okay. I see. I see. I see. See what you actually wanna do that
MattGPT says: then you'll filter.
MattGPT says: is, there's not. It's not dot lock right? Because it's not the index, right? So it's actually you just want to do
MattGPT says: where you've got dot lock, replace that with double equals
MattGPT says: right? Equal equal to delete the lock and the square brackets. Open square brackets. Yeah.
MattGPT says: And there we go. Okay, okay.
MattGPT says: so run that, let's just see if that works.
MattGPT says: There we go. Okay. So that should have just a few lines in it, right? The and that date should all be yeah, it's got. It's got this 16 rows or something like that.
MattGPT says: Okay? So that gives you just the shootings in October 2023. Okay, so that to right? And so just what's going on there is
MattGPT says: when you say shooting underscore df, and then select date underscore. Ym, right? You're pulling out that column, and then you're doing a true false. You're doing a test on each row
MattGPT says: whether it equals the string. 20, twenty-free dash 10. Okay, now, what you want is the inverse of that. So if you put a minus there right and then capture that result. So at the beginning of that line.
MattGPT says: then that's when you want to say, shoot the beginning of that line. Yeah, put in the name of the data frame the the subset. So yeah, right at the beginning. No, right at the beginning.
MattGPT says: Beginning of that. Yeah.
MattGPT says:  hang on.
MattGPT says: Okay. This is. yes, equals.
MattGPT says: Yeah. Yeah. But this name here, this name here should be this right here.
MattGPT says: okay.
MattGPT says: You got it. There we go.
MattGPT says: Okay, thank you. So, yeah, yeah, just make. So I see, I saw what you're doing. But when you say, dot lock, okay, what that is doing on a data frame is that is actually
MattGPT says: using the index. Okay? And so the index in that data frame was not the date month, date, year month column. Okay, you could set the index to be that. But we just have the plain index, which is the numeric one.
MattGPT says: So what you do is, you create a filter that looks like you've got here. Okay? So go to the data frame. This is the column that we're interested in.
MattGPT says: row by row. Test each value in this column to see whether it equals 2023, dash 10, the string, 2023, dash 10. It's going to be either true or false when you say minus that terms. So most of the rows are going to be false, because there's only 16 rows which are in
MattGPT says: October 2023. When you say minus that switches, the false is the truth. And so then you're selecting everything which is pre which goes up to September 2020.
MattGPT says: Okay.
MattGPT says: Anyone else got any questions.
MattGPT says: okay, do you want to share your screen and we can have.
MattGPT says: Okay? Can I take the average number of interns each month? And
MattGPT says: So first, so yeah, so if we break it down right? So this is
MattGPT says: is is asking you to to generate
MattGPT says: a monthly mean number of incidents in twos in 2 subsets of the whole data frame. Right? So the first thing to do is to subset the data frame. Okay? So the the distinction is between before 2020 and after 2,019, right? So the first thing I'd do is define a filter.
MattGPT says: Okay? And just call it like, I don't know.
MattGPT says: filtered 20 underscore 2020 or something.
MattGPT says: And so
MattGPT says: we're working on the shoot to set 2020 free data frame. Okay? So you're gonna select that
MattGPT says: data frame.
MattGPT says: Okay? Now, remember the fields that we have. We've got date underscore. But that's you know the whole string we want to. We can just use the year  column here, right?
MattGPT says: Which is just an integer. So you'd say.
MattGPT says: yeah, that's right. Double equals
MattGPT says: well, let's see. What do we want to do? No, we want before 2020, don't we? So do less than 2020.
MattGPT says: I do
MattGPT says: less than right.
MattGPT says: well, it depends which way you want to do it. But let's do less than 2020 right?
MattGPT says: So why don't we call this filter filter underscore, pre, 2020 is a good kind of name for that right? Filter. Underscore. Pre, 2020. Yeah, perfect. Okay, so can you see what that's doing? So in your data frame, you're pulling out the year column and you're saying, row by row.
MattGPT says: Does the year value is the year value less than 2020.
MattGPT says: True, false. Okay, so that will give you a filter that will split the data frame. So then, your next step like you you are there now. Is that to calculate the mean?
MattGPT says: The steps are
MattGPT says: subset the data frame by the filter you've just defined.
MattGPT says: Alright, so you can delete that sum greater than
MattGPT says: So select the data frame right? So shoot to set 2020
MattGPT says: free alright.
MattGPT says: and then square brackets. oops hanging. Go to typo there.
MattGPT says: 2023. That's it.
MattGPT says: Okay, and then square brackets, and then you you can use a row. You don't. You're not selecting a column. You're using row filtering. So you put your filter there we go right. So what that will do is that will give you all of the rows where the date is less than 2020 in the year.
MattGPT says: Okay? And so now we wanna, we want to group by month. Okay? So
MattGPT says: instant average instance per month. So that so you could do 2 things that right? You could just now select your date
MattGPT says: you could select your date year, month column.
MattGPT says: So delete that. There, yeah. So yeah, that's good. No, no
MattGPT says: square brackets, yeah. And then have in square brackets and in quotes date. Underscore. Ym.
MattGPT says: okay. But it's a field name. So you want it in in quotes.
MattGPT says: yeah, there we go.
MattGPT says: Okay, good. And then, if you did, a dot value counts.
MattGPT says: Okay, open close premise. Remember, that gives us our tally of stuff. And then if you did, dot mean on that. okay. So that's going to first the first bit, the value counts gives you a
MattGPT says: yeah, okay? And then if you take a look at what that looks like right? So if you run that cell  see what's it saying?
MattGPT says: Name?
MattGPT says: Oh, you missed the underscore df, in both of those places right at the end. That. So set 2023 underscore. Df.
MattGPT says: yeah. And then the same in the next line down.
MattGPT says: Okay? So you can see that it's done the average number of shootings since per month. What's that?
MattGPT says: So then your post, you can copy that whole line, basically right? Your post one is going to be the same
MattGPT says: thing. You don't need the filter. You don't need the filter. Right?
MattGPT says: Yeah.
MattGPT says: okay, you got it. Yeah. Okay. So then it's going to be post 2,019.
MattGPT says: He's just gonna
MattGPT says: post
MattGPT says: Is what we're asking yet. That's it. And then do a minus on the filter.
MattGPT says: And for
MattGPT says: no, it's going to be. No, no, it's the same filter, right? But you just need to inverse it right? So you you only need one filter. You could define 2 filters, but because these are the inverse of each other and just put a minus there.
MattGPT says: Okay?
MattGPT says: And then that's everything in there. Right? Okay.
MattGPT says: alright. Let's see if that gives you the correct. Yeah.
MattGPT says: there we go. Thank you. Okay, does that make sense? What's going on there? Right? So you just think this, just go over it. And and I'm people that are watching to see that it's that you're gonna
MattGPT says: first
MattGPT says: think about. Okay. So what have I got to do. I've got to calculate 2 averages. Okay, so and the turning point is 2020. And so I need to subset my data frame by year.
MattGPT says: so they you can't, you know, use, you would try and write it all in one line. You could do that, but in this case the you know, break it down and say, first, define a filter that gives me those 2 subsets when it's something like this which is before and after. You actually only need one filter, right? So to pick the easiest one there is just to say less than 2020 and then
MattGPT says: the
MattGPT says: to calculate the mean it's going from left to right. There. You've got to select the data frame.
MattGPT says: subset the rows based on your filter.
MattGPT says: and then count up the number of rows in each value of date. Emo, okay, because what's what you're going to end up with. There is a table where it's got like
MattGPT says: 2,015 dash 0 one. How many rows have that date? Year, month, value
MattGPT says: 2015, dash 0 2. And all the way through to 2019 dash 12. Okay, so that gives you a distribution of the number of
MattGPT says: incidents per month, and then what the question is asking you to do is just take a mean of those right. And so this is what's nice is that you just keep adding functions to the right of each step.
MattGPT says: Okay, select the data frame.
MattGPT says: filter it by using my filter so that I only get rows where the year is less than 2020 then in that. So in each of those 2 subset, or in that subset, select the date year, month, column.
MattGPT says: and count up the number of rows for each unique value there. Okay, so that's gonna be strings that start from 2015, one and end at 20
MattGPT says: 19. Dash 12, right? So January 2015, all the way up to
MattGPT says: December 2019, okay? And then you're gonna get a table that gives you each of those values plus the counts.
MattGPT says: We just want to get the mean of that. So if you stick the dot mean on the end.
MattGPT says: then you've answered the question, right? So this part selects the filter.
MattGPT says: Yeah, so yeah, the column. That's right? Yeah.
MattGPT says: Okay. So if you want to understand, kind of what's going on right, just add a blank cell. I add a blank cell sort of above, there or below there, right you can.
MattGPT says:  if you go over to to your icons here.
MattGPT says: you want to select that
MattGPT says:  you see. Click on this little icon right here.
MattGPT says: I'm just trying to draw on your screen. Can you see my annotation? Yeah, so select that though I can't write that.
MattGPT says: Okay, like, it's just for just for illustration, for so now take each part in turn. So start with this part here.
MattGPT says:  let's see? So take
MattGPT says: rolling thing.
MattGPT says: Okay, yeah. What I want you to do. Sorry. It's just take.
MattGPT says: Take that first part there, copy that.
MattGPT says: Okay, and then paste it into your black cell there.
MattGPT says: Okay, so step one.
MattGPT says: You see that that gives you. If you scroll down to the bottom that gives you 6,821 rows. Right? Those are all rows where the year is less than 2020.
MattGPT says: now add, in the next part of that which is the square brackets?
MattGPT says: yeah. And that's in quotes, remember here. Okay, so then you're getting you're getting just that column
MattGPT says: again, you're getting 6,821, right? And so you can see.
MattGPT says: I mean, it's out of order. But it's it will start from 2015 0 1.  okay. So now you put in dot value. Underscore counts.
MattGPT says: Okay, and then run that. You see that you get that.
MattGPT says: and that is ordered by frequency. But like that's fine, but you can see that you get each value, each distinct year. Month value is then counted up. And so what the question is asking you is like, what's the mean of that second column?
MattGPT says: If you had taught me.
MattGPT says: then you got the answer, okay.
MattGPT says: so that makes sense sort of the step. Yeah sort of across there
MattGPT says: makes sense to know like that cool
MattGPT says: anyone else but questions. I'll
MattGPT says: and it may be that you like, I want to use this time, you know, to feel to any questions. So if you've got an answer and you like, well, I managed to get it to work, but I'm not exactly sure
MattGPT says: what's going on. I'm happy to do that sort of step by step, breakdown
MattGPT says: or explain stuff more generally.
MattGPT says: Okay. Hi, everyone can everyone hear me? Okay.
MattGPT says: Great.
MattGPT says: excellent thanks.  okay, good to see all.
MattGPT says: So
MattGPT says: let me just share my screen. What I wanted to use today's time for was just to sort of
MattGPT says: So any questions that people had about like
MattGPT says: oh, stuff that people wanted me to go over in terms of like
MattGPT says: what we've covered so far in doing their analysis
MattGPT says: and data storytelling, using pandas and the various plot stuff.
MattGPT says: And then we had.
MattGPT says: oh, yeah, Zach.
MattGPT says: Yeah, I'm still working on trying to figure that out, so I'll try and get that posted and let you know to tip when you can test it. But I'm not.
MattGPT says: I yeah, I haven't been able to post that all of the feedback yet.
MattGPT says: And so everyone. Yeah, you'll see that I posted Simon for the other day. I think, Saturday and let you know. And so you've got until tomorrow
MattGPT says: to finish that
MattGPT says: if you need a little bit more time, and you want to eat into your full break. You're welcome to do that if you just let me know
MattGPT says: but probably good idea to try and get it done before that. So you can actually have a bit of break. Okay? So
MattGPT says:  so yeah, just a few things I wanted to mention. We can look at
MattGPT says: a few of the plotting things that we were looking at in a bit more
MattGPT says: details.
MattGPT says: you know, I'm open to questions. If you want to work on assignment 4. And while I'm here to sort of answer questions, we can. We can do that
MattGPT says:  something that we may pick up again.
MattGPT says: So you remember, when we were looking at the
MattGPT says: it was in 10, I think.
MattGPT says: Yeah. So we were looking at that shootings database right?
MattGPT says: And one of the questions was about officer involved shootings
MattGPT says: and
MattGPT says: we saw, and that group found out that when Officer involved, when there's an officer involved
MattGPT says: in a shooting. Then. seems like we've got the race
MattGPT says: and ethnicity information missing. So some of the demographics had it had gender in there. I had sex in there, but didn't have
MattGPT says: race, but it's there for everything else, and we're sort of wondering what that is.
MattGPT says:  looking around a little bit more I noticed a couple of things that were interesting. One is that
MattGPT says: the Philly police department has an official complaint process.
MattGPT says: So if you want to complain against an officer or something.
MattGPT says:  There are different ways of doing it, probably, but this one way is that you can go to this.
MattGPT says:  you can go to this website, and you can fill out a complaint form.
MattGPT says: Now, what's interesting when you
MattGPT says: and when you look at this is that this is, if you were making a complaint, then you had to put in your information and some address and some identifying.
MattGPT says: They want age. They want some specific demographics just when you are making a complaint.
MattGPT says: Okay? So
MattGPT says:  not sure if these are actually required or or not. because they don't seem to be consistent necessarily where the and tried to fill out the form. Okay, but so.
MattGPT says: but it's it's interesting that you know. You have to give these identifying demographics.
MattGPT says: And then there is
MattGPT says: a kind of race category which is confused with with Stephanie or the Hispanic Latino.
MattGPT says: one that usually kind of is broken out as a separate one there. So they're collecting the race potentially of someone who makes a complaint.
MattGPT says: And then you would put in the information about who, specifically, you were complaining about right? And so you
MattGPT says: ideally have an officer badge number and a license plate and those sorts of things.
MattGPT says: And then you'd put those in there, and you can put in demographics about the officer anything that you remember there, and then you've got those same categories there.
MattGPT says: And then you can put in the details there.
MattGPT says: So
MattGPT says: If you look, then, at the complaints against the police
MattGPT says: databases that are available on open data. Philly can see that they have
MattGPT says: the complaints against the police.
MattGPT says:  and then they have the findings related to complaints. Okay, so this will be complaints going to come in, and then they have
MattGPT says: complainant demographics in a separate Csv. And then they have sort of like a history which seems to be related to
MattGPT says: the whole process of complaints. Right? So, on the one hand, I'm like, this is cool. Yeah, this is really good, that this is part of their account availability process and making. They're keeping this data up to date, you know. So compared to a lot of the
MattGPT says:  departments. And
MattGPT says: you know that we're seeing governmental kind of departments that we're seeing on open data portals and particularly open data. Philly.
MattGPT says: the Philadelphia Police Department are keeping the crime database up to date. And the Prime Minister database seems to be up to date and you know, there.
MattGPT says: hosting
MattGPT says: easily accessible data sources so that you can make data stories out of it, which you know, II think it's all is already good. But coming back to sort of the question of
MattGPT says: who collects data and what they collect and why things are collected.  I just found this kind of interesting in the light of the withheld, or we're not sure exactly of the reason why, when there's an officer involved shooting.
MattGPT says: we don't see the victims demographics.
MattGPT says:  and it doesn't include
MattGPT says: at least in that database any information about the officer?
MattGPT says: but when it comes to making a complaint you can't just make an anonymous complaint easily, it seems, without including some demographics. And what is the relevance of demographics or reporting who is complaining? I'm not entirely sure. So
MattGPT says: just wanted to call attention to that. I think it's it's interesting. And we could. It's the kind of thing that as we start to think about
MattGPT says: data projects to do
MattGPT says: what you'll what you want to do with kind of starting to think about project is sync about where you can have
MattGPT says: 3 or 4 different data sources that you can bring together, that you can link together because
MattGPT says: you know, we've got location there. When you look at the metadata for these complaints, they do seem to link
MattGPT says: sometimes to where there's a complaint. Id which goes across all of these. But
MattGPT says: there may well be that they can be linked to a specific crime incident, and certainly in a location. And so there's number of different ways that this these data could be brought to bear with looking at where there are shootings or where there are crime incidents in the in the other sort of database.
MattGPT says: I'm aware there are other things at the neighborhood level.
MattGPT says: So
MattGPT says: yeah, I wanted. I wanted to call attention to that.
MattGPT says:  Let's see. And then the other thing.
MattGPT says: we were. We we went over Some of the different plotting options that we have right? And we've been using and continue to use
MattGPT says: map plot lib at the base level.
MattGPT says:  and
MattGPT says: We also then started to look at Plotley. And so Plotley express. In particular, is this
MattGPT says: library that I'd encourage you to take it. Take a look at
MattGPT says: And if you look here at the basic charts.
MattGPT says: it's a interactive. It produces interactive plots. Okay? So I think
MattGPT says: below the hood, it probably uses something like d free Javascript library.
MattGPT says: To produce these things, and that that, you know it has all the basic sort of level plots in it. Scattered pops.
MattGPT says: And
MattGPT says: the key thing. With all of these.
MattGPT says: you know, they're very. It's very easy to use.
MattGPT says: if you just import plot the express and the normal alias for that is Px. Then
MattGPT says: it's Px dot function, which is then most of the basic plot types. So you've got bar, you've got lines
MattGPT says: area for area plots. And then you've got scatter, plot.
MattGPT says: and at the simplest you can just give it to match lamp sequences or lists
MattGPT says: of data X and Y, or if you provide a data frame.
MattGPT says: then your X will be a string, which is the name of the column
MattGPT says: that is, gonna be on the X axis and the Y axis.
MattGPT says: and you can see that
MattGPT says: you get a plot, and then, by default, it will give you
MattGPT says: hover over each point or bar, or whatever your plot type is.
MattGPT says: and show you the data. Okay? And then there are a bunch of extra
MattGPT says: arguments that you can give where you can control what is shown in your hover. If the hover is shown at all extra pieces of data that you put in there.
MattGPT says: Okay, so.
MattGPT says: here's an example of doing that with a data frame. Okay? So
MattGPT says: so standard, the irises data frame the irises data set just 3 different types of
MattGPT says: viruses in it, and the length of the
MattGPT says: pedals, and and so on.
MattGPT says: you can see you can just put in the data frame and then specify with strings the X and the Y
MattGPT says: and that gives you an interrupted one. So again, this this is II like using this for exploratory data analysis. So in your notebooks, when you're trying to figure out patterns in your data.
MattGPT says: Simplest thing to do, a static plot. And that comes straight out stuff. But you can use partly express
MattGPT says: if you want to have something where you're actually gonna do some and particularly if you introduce a third variable like here. Okay, so
MattGPT says: got the X and the Y the same here. And then you can specify color as a third variable. Okay? And then, as soon as you do that you'll automatically get a legend up here. and then you can use size potentially for a full variable.
MattGPT says: Now with these sorts of things once you get to.
MattGPT says: you know, with scatter plots or other sorts of plots.
MattGPT says: usually the extent of what you can kind of visualize and take in as probably free variables. But here, potentially, you can find patterns with before. And the color nicely allows you to see. Okay, yeah.
MattGPT says: this particular species is really separated from the other 2, and the other 2 are
MattGPT says: kind of mix, but there is some sort of pattern here
MattGPT says: where the green seems to be a little bit higher, a little bit larger, with its sepulm and its width and then
MattGPT says: we've got the actual.
MattGPT says: We've got pedal length as the size of the point.
MattGPT says: And so again, you can see, really, just see a species difference, and maybe a bit of a difference between these 2. But between the blue and the the other 2.
MattGPT says: So
MattGPT says: you know you can do. You could certainly do this as a static plot.
MattGPT says: the syntax, for this is.
MattGPT says: I think, really nice and straightforward, where you just give it the data frame, and then you can just give your your, you know you're doing your mapping from your data to the the Graph Dimension
MattGPT says: Plot on the X axis. The Y axis use the color hue.
MattGPT says: the size of the points. Okay? And then you see
MattGPT says: here where we have hover data as a an additional
MattGPT says: argument. Then we give it a list
MattGPT says: and you can see what that's doing is, it's actually adding an extra piece of data to what is already there in the hover data.
MattGPT says: Okay, so you can potentially have other pieces of information that you can't plot in some sort of way, but you can add them into your hover data
MattGPT says: so that you can and
MattGPT says: look at patterns like that. And then, if you look in the documentation, there's ways to actually make this a little bit
MattGPT says: prettier than just, you can specify a sort of like an S string or a format string to show you know how you want these to show up.
MattGPT says: but by default it will just give you the name of the the variable equals, and then its value.
MattGPT says: Okay,
MattGPT says: And so
MattGPT says: where you've got, you know, in this case here, and then it's the other payment to notice here is that
MattGPT says: well, using color here. Species species species is a categorical, variable, right? So it's nominal variable here.
MattGPT says: But you can just pass it, a, an, an another variable in here. This is quantitative, variable right, which is pedal lamp
MattGPT says: or numeric variable. And so then you get automatically.
MattGPT says: scale. Okay? And you get a sort of a heat map for the situation.
MattGPT says: So it's really cool again. You know, it's I would just stress. Like, I think this is fine. You definitely should play around with this, use this but think carefully about before using it in communication and expecting people to make sense of these and to be, you know, I wanna hover over every single point and look for things. Okay?
MattGPT says: But at the same time. The reason I want you to get used to it.
MattGPT says: Play around with it is that
MattGPT says: We're gonna look at a library, called Dash.
MattGPT says: Let me come back after the break.
MattGPT says: and
MattGPT says: That's the best way to point you to my one.
MattGPT says: that is built on top of Plotley, so that uses Plotley as
MattGPT says:  the
MattGPT says: underlying plot library, and then it allows you through. Pipin
MattGPT says: also allows you, if you're an art person, to do it through our. It's a kind of alternative to using shiny. If you've done any our programming and use shiny
MattGPT says: to very quickly build up a dashboard using these components. If I just want to mention that I'm not going to look at that today. But
MattGPT says: so yeah. And then so you you can see like. So what I really like about this is just how straightforward it is right just to have this as an import. We've got a data frame. And then it's just one function. Call
MattGPT says: at least to set up the basics with a whole bunch of different arguments that you can specify right where you're doing a mapping from
MattGPT says: column to some visual back. Okay?
MattGPT says:  alright, yeah. And you can see that it shows you actually in this documentation how you would use it in dash. Okay? And so what that gives you is an application which can then have
MattGPT says: multiple panes, which are like dash dashboards, and you can add in width widgets like this, where you can do filtering.
MattGPT says: so it gets a little bit more complicated, but not that much more complicated. We'll talk about this and how to set up these widgets and to set up the update functions.
MattGPT says: But in
MattGPT says: sort of 10 lines of code, if you have a plot already figured out from your data set, you can turn that into an interactive dashboard.
MattGPT says:  so yeah, I'd encourage you to to spend a little bit time with this documentation
MattGPT says: and with some of the data sets that we have or data sets that you're interested in. And then
MattGPT says:  you can see that, you know, for various types of plots.
MattGPT says: it allows you to do all sorts of potentially useful, but also potentially confusing. Things like
MattGPT says: this is a scatter plot of 2 variables on the Irs data set where we've got
MattGPT says:  this April length versus the width on the Y axis. And so we've got 2 variables, continuous variables. And so the scalar plot is the right one to use.
MattGPT says: But then you but you may have seen this before. You can add in additional plots. Across these axes to look at the distribution. So this is a histogram
MattGPT says: of the variable on the x-axis. So the length
MattGPT says: okay? And so it's an interactive histogram. You can see that you get the bin width
MattGPT says: and then the council.
MattGPT says: So it's and then here you could you sometimes see, this, too, is that you have a histogram flipped around onto its side, and you could have one here.
MattGPT says: But there's also something called a rug plot. and what a rug plot does is show you
MattGPT says: where all of the points are without.
MattGPT says: in in your data.
MattGPT says: and instead of having count being shown by height. Then it's trying to show each of the individual points. And so you can see where this line gets bigger.
MattGPT says: Then there are more points at that particular value.
MattGPT says: Okay, so it's another visualization, basically of distribution.
MattGPT says:  I would say, never can. I think of a good place to use this in a data story, right where you're trying to communicate to people.
MattGPT says: Here's some data that we've been investigating. Here's the patents here. This is just most people gonna look at this and be like Whoa, I don't know what's going on. Maybe potentially, maybe a scatter plot. You can understand histogram. You may have a reason to show if you
MattGPT says: if you really explain what's going on, what people should be
MattGPT says: should be saying.
MattGPT says: so you know, just because these plots allow you. These libraries allow you to do it doesn't necessarily mean it's always a good idea. But as you go through you can see
MattGPT says: it's a really nicely thought out.
MattGPT says: way of of doing these plots. And it's, you know, it's just perfect for doing exploratory data analysis where you want to be able to look at the points and look for patterns.
MattGPT says: Okay, so
MattGPT says: yeah, that's plot the express that I would encourage you to take a look at
MattGPT says: and then any any questions that anyone has on anything related to that?
MattGPT says: If you like, what you know, what I'm thinking that we can do, or what I would encourage us to do in this time is like.
MattGPT says: if you want to look at assignment 4. Okay, and if you haven't
MattGPT says: started working on it. Yeah, if you want to start working on it now.
MattGPT says: and be able to ask questions, then go ahead and do that if you're already done with that, or you want to actually
MattGPT says: do something else more with the plotting.
MattGPT says: then I'd say.
MattGPT says: that's you know you can spend some time
MattGPT says: with
MattGPT says: various data sets. Work your way through some of the examples in the documentation.
MattGPT says: I'm happy to talk to you about that.
MattGPT says: So assignment 4. If you haven't had a chance to look at it yet, is
MattGPT says: building on what we've done before. Okay, so there are free related open data sets.
MattGPT says: One of them comes from the Acs, which is this big relate to the census. And so it kind of comes around definitely every 10 years, but then they do intervening years where they do statewide
MattGPT says: and what countrywide surveys within census unit, you know. So they try and do their sampling a stratified sampling, so that within the different within a city within a area.
MattGPT says: they try and make sure it's represented at the different census units so that it can be tied into the
MattGPT says: the the decade level census census data.
MattGPT says: And they ask multiple questions and they do it in a stratified way. So not, you know. It's not that everyone gets answers in the sample, gets answered every question, but they stratify them, and they cover things like health. And
MattGPT says: first.
MattGPT says: you know, demographic behavior, metrics incidence and cancer and heart disease and those sorts of things.
MattGPT says: and then they also ask about physical activity and
MattGPT says: other things in daylight diet, and so on. And so one of the survey questions in there is this one about
MattGPT says: a
MattGPT says: whether you commute regularly by on foot or by bike, and then they aggregate these data and release them at that level with the percentage of
MattGPT says: individuals within a State, or sometimes within smaller units within States
MattGPT says: know if it goes because it goes down to the census level
MattGPT says: to meet that. And so that's what this, the main, the the bulk of the questions, are actually looking at this particular data set, doing some subsetting, doing some analysis.
MattGPT says: One of the years that is covered in the data set is 2011. It goes from 22,009 to 2,019, I think.
MattGPT says: the data that's in this particular sample. And then the traffic fatalities contains the number of traffic fatalities by state
MattGPT says:  in that particular year. So you can get a sense of, you know the frequency of traffic fatalities at the state level.
MattGPT says: Okay? And so. And then the first one and the third one is another. Acs question, that gives responses to a question about physical activity levels.
MattGPT says: and it's a 4 year sample 4 year sampled within this time. Period.
MattGPT says:  and I think it relates to from what I remember looking at, it relates to
MattGPT says: the portion of the population. They get more than the get. Get 150 min of moderate activity a week.
MattGPT says: Just sort of Cdc guideline, right? So you can see how the the idea here is that you'll exploring these data sets, and I'm asking you some questions to sort of
MattGPT says: subset them, to create some means to do some plots, to look for some patterns across them individually? And
MattGPT says: all of these questions is put into practice. What you've
MattGPT says: what we've covered so far with using pandas right? And there's some plots. And then the last part is for you to write a data story, blog post
MattGPT says: and
MattGPT says: these separate notebook. Yeah.
MattGPT says: And there you've got 3 data sets that you can bring together to make a story. Okay? And you can think about maybe some ways outside of these questions, there's less analysis done on the second and third data set, but you could do a little bit more, but certainly you can see how you could hone in on specific
MattGPT says: year. A specific area of the country or a specific state, and try and see if you can find some interesting patterns.
MattGPT says: Okay? And that's what you're going to do to turn into your story.
MattGPT says: So that's assignment for
MattGPT says: And yeah.
MattGPT says: and so yeah, I'll turn it over. Basically.
MattGPT says: Now we'll use the rest of the time.
MattGPT says: for if there's questions about stuff.
MattGPT says: just jump in. Ask me a question. I'm happy to, you know. You can share your screen. I'll try and help you out with those
MattGPT says: and I would suggest, yeah, spend. Spend the time working through assignment 4. If you want to work on that.
MattGPT says: If you want to actually do a little bit more exploration of Plotley Express in particular, then you can do that. Yeah, Max.
MattGPT says: Yeah, hang on. Let me just
MattGPT says: Take a look at that.
MattGPT says: Okay, so let's see.
MattGPT says:  oh, I see. Okay.
MattGPT says: yeah.
MattGPT says: So this is something funny that we haven't covered before. But you can. Yeah, I can see what you're doing there, which is putting the years in a list what you want to do. If you want to do that. Then you need to do a set function. Okay? And there's one. So after the wh, where you put your double equals, you want to put.
MattGPT says: you want to delete that. Okay? And there's a function. Go back one. Then dot
MattGPT says: is in. ISIN, and then in parentheses.
MattGPT says: put your list. Yeah.
MattGPT says: okay, so that that will actually do you, I think what you were trying to do there. Yeah.
MattGPT says: function.
MattGPT says: And you can put anything in that.
MattGPT says: so, the look function is for the is for the index. Okay? And so,
MattGPT says: what? Yeah. So
MattGPT says: plus
MattGPT says: another option. So can you put in a blank cell, actually say, above where you are?
MattGPT says:  so let's just just do take a look at as a Acs free. Underscore. Df, okay, just just fine. Just run that. Okay? So you can see that the the row index is just
MattGPT says: an in integer right now, going from 0. Right? So if you're gonna use the log function like you do dot loc.
MattGPT says: sure. And then you have square brackets, and then in there you can put an index number right? So you could put like, or you could put like one comma 7 comma 400.
MattGPT says: Just put it? Yeah, right? And so that would get you those rows right?
MattGPT says:  sorry you have to put those in another set of premises. Another set of square brackets. Yeah.
MattGPT says: yeah.
MattGPT says: there we go, right? So you can pick out rows like that. So that's the that's the concept. With that. So you can. What you could do then is so now delete that down, back, down to the now, if you do acs free.
MattGPT says: Yeah, if you did set underscore index.
MattGPT says: and then, in prem season, in quotes per year.
MattGPT says: So parentheses
MattGPT says: Yeah. Oh, great. Okay. So I think that was my Internet. Now, now, run that.
MattGPT says: Okay, see what? So what that does. This is sometimes useful to use. This is that
MattGPT says: sometimes you you actually do want to use one of your variables, one of your columns as an index. Okay, so now, if you did dot lock on that after that.
MattGPT says: And so yeah. So that was probably what it was getting at there like something like that would get you
MattGPT says: got those right and then depending on what you want to do like. Sometimes you would then say, Okay, II want to reset my index because I want to group by it so you could do a dot reset underscore index at the end of that.
MattGPT says: and then open close parentheses.
MattGPT says: Okay? And then see, you're back to. But you subset it. Your data frame. Now, okay, okay, so that's you, you, that's another pattern that I haven't really shown you yet, because I think filters in some ways are E,
MattGPT says: yeah, somewhat easier to to get at but this is a really common thing that you see, which is
MattGPT says: where especially things like dates are really good as indexes, because you can actually then get ranges between dates and stuff like that so much? Yeah.
MattGPT says: Yeah, I mean, yes, you wanna you, you wanna do that right? So
MattGPT says: do I have any examples around? And
MattGPT says: I mean.
MattGPT says: can just quickly sort of create one. But yeah, what you want to do is just add some cells in here and put it so like, you know. So whatever your title is
MattGPT says: like?  I don't know.
MattGPT says: Why do we? Where do we like to work?
MattGPT says: And
MattGPT says: does it shorten all eyes? Right? So say, you're gonna kind of come, come with something like that right. And then.
MattGPT says:  you know.
MattGPT says: working with
MattGPT says: state level data.
MattGPT says: From the and all
MattGPT says: meaning
MattGPT says: And now.
MattGPT says: etc., right? So you put all that together. And then I'd say, you know, then you would have a
MattGPT says: because this is in a different
MattGPT says: notebook, then, you know, import the things that you need
MattGPT says:  and you can just copy these across from
MattGPT says: another thing that
MattGPT says: my plot.
MattGPT says: Okay? So and then, you know.
MattGPT says: send the data. But
MattGPT says:  okay. So then you'd put your cells in there and then
MattGPT says:  in between of these, then, you know I would you'd have some sort of output, and then, you know.
MattGPT says: do some level headings.
MattGPT says: Thank you. Observations.
MattGPT says: etc., etc. Right? And then
MattGPT says: so this this. Yeah, you you just want to interleave. Mark down cells with code cells to build your story.
MattGPT says: But
MattGPT says: and does, does that kind of help
MattGPT says: counsel your question a little bit perfect? Thank you.
MattGPT says: Yeah. So that's the way. So do it. So if you you know from the assignment you're like, Oh, that was some cool stuff I found when I did question 6, and I want to use that again.
MattGPT says: Then, you know, you can figure out well, what do I need to bring across from that
MattGPT says: copy that over to here. But the idea of the blog post is stands on its own, and someone can read through it, and that you, the key thing is to make sure that it reads well, right? And it's a story. So this is giving us sort of a data story.
MattGPT says: Yeah.
MattGPT says: and often like people that say, well, how how long should it be?
MattGPT says: You know.
MattGPT says: it depends what your you know what your story is. okay. it's, you know
MattGPT says: something that takes sort of like 4 or 5 min to read, probably is a good length. If you're thinking about like, it's a medium blog post where it says, this is a 3 or 4 min. Read or 5 min read
MattGPT says: coming along those lines is what you're aiming at.
MattGPT says: But really you're trying. Don't just reproduce, like the analysis of each data set. What you're actually trying to do in the blog post is You've done all that analysis and then think, okay, so what's an interesting angle? What's the story that I can tell from this? Right? And so that's where you know
MattGPT says: you can think about like, are there one or 2 really interesting patterns that I find here? Or is there a question you can just start off with and state that question which is like, you know.  if you can mute my bike, then
MattGPT says: yeah. you're more likely to be healthier. And so.
MattGPT says: you know, we've got these metrics, which is, is there an association between how many people commute or walk to work, and the amount of people who are the percentage of that populate of that State's population that are meeting some health requirements. But then the flip side of that is like community by bike is dangerous, right? And so
MattGPT says: potentially, you know, that's why you've got the traffic excellent data. And you can bring those together. So you're trying to think about like, what's the
MattGPT says: what's an interesting story that I can tell by bringing these bits of data together.
MattGPT says: And if you want to, and if you you know, you find something else about one of the States that you want to bring in.
MattGPT says: that.
MattGPT says: you know, annual rainfall.
MattGPT says: You could probably quite easily find some annual rainfall data that is at the state level.
MattGPT says: and that could be another reason to, you know another thing that explains us lower than the average amount of commuting or walking
MattGPT says: to work and stuff. So
MattGPT says: that's really what I'm looking. You know what we're looking for. There is in the in the blog post this fee just to try and
MattGPT says: come up with something creative with the data sets.
MattGPT says: right? And you don't know. Yeah. So the test is.
MattGPT says:  adding together the 2, 2016 and 2011, and then dividing by 2. So you did that up above, right
MattGPT says: in your cell. But you just got the one test.
MattGPT says: yep.
MattGPT says: kind of
MattGPT says: Okay, can we see? Can I just check where you did where you defined Acs. 11.
MattGPT says: That 2811. Df, that's the previous question. Probably right?
MattGPT says: and if you, if you do like a run, all
MattGPT says: does it? It does. I'm just wondering if it
MattGPT says: Reynolds. Yeah, yeah, let's try that.
MattGPT says: Hold on.
MattGPT says: Oh, yeah, there you go.
MattGPT says: Okay. So something weird happened in the order there. Right? So I think what somehow you had gone back up and run the cell before you did the mapping to Alaska, and so acs free underscore. Df, right before you, added Alaska into the Far West with far West thing. So that's the that's the
MattGPT says: the
MattGPT says: the difficult thing about notebooks, right? They're great for doing this sort of stuff, but like you jump around and then you suddenly like, Oh, I'll go back up and I'll run something from question 2 right? And that happens to overwrite.
MattGPT says: You know, that's before you've done the mapping right then. That's so. That's what happened right now, you see, like so always, if you like. You know, I think your intuition was right, which is like this looks right. The test, the passing. Why is it wrong.
MattGPT says: If you get that, then always. First thing to try is like, let me try. You can even just do a run all but like Colonel Restart, and run. All is the best thing, because that will clear out. Everything can run it from the top.
MattGPT says: I know it's super frustrating. When that happens.
MattGPT says: Can we do the same thing that we just did. So if you go up to
MattGPT says: Kernel and restart runnel.
MattGPT says: Yeah. So
MattGPT says: let's see.
MattGPT says: yeah, that code looks right to me. But
MattGPT says: there we go.
MattGPT says: Okay. And that's that's still.
MattGPT says: Not
MattGPT says: a
MattGPT says: is it? Do you think it's a problem with the brackets, or do they look fine? No, that's fine
MattGPT says: that
MattGPT says: I'm using, Mac, which is the right one.
MattGPT says: Can you just put a a blank cell in underneath that current cell? Let's have a look.
MattGPT says: Okay, so let's do A/C,
MattGPT says: that's free underscore df.
MattGPT says: and then in square brackets region.
MattGPT says: Let's just see while we're getting there.
MattGPT says: Yeah. Okay.
MattGPT says: So for most of the.
MattGPT says: And can you put in state in there instead of region to see what's happening.
MattGPT says: Okay?
MattGPT says: Oh, I see. Yeah, okay, I see what's happening. Yeah, can you go back up.
MattGPT says: yeah, I gotcha. Yeah. Go go back up a few lines, a few to the question where we define state to region. That's right. There, isn't it yet. So run that cell again.
MattGPT says: I see what we've done. Yeah, okay, I got it. Okay. And then, now you can scroll down to where we were
MattGPT says: now, instead of instead of creating a whole new dictionary, which is what you've done there. So now, what's happening is that only only lines where there's Alaska is getting is getting a value. Everything else is getting na. So what you want to do is is just add in a key value pairs, and don't define a new dictionary. So delete that curly brace and equals
MattGPT says: okay? And then and that one, yeah. And then it's gonna be delete that back onto the yep square brackets.
MattGPT says: Ak, okay, close, square brackets and then equals instead of a colon there.
MattGPT says: Oh, okay, you got it. So what that's doing is actually, yeah, there we go. Oh, okay, that makes sense. Sorry. Yeah. So
MattGPT says: yeah, what you had done is that you just defined a completely. You redefined the Mapping dictionary, so that only had one state in it. so that what was happening was that every other state, apart from Alaska was getting a null value right, because there was nothing in the mapping dictionary.
MattGPT says: Does that make sense? That makes sense? Thank you.
MattGPT says: Yeah. Good good question.
MattGPT says: let me just find it hang on. And
MattGPT says: so, after Katie's question, I was just going back to look at
MattGPT says: previous
MattGPT says: things. Now I'll show you some of these that people did.
MattGPT says: So.
MattGPT says: you know, like, I said, keeping, you know. Come up with a with an angle, or frame it in some way title and then have an intro
MattGPT says: framing stuff. What's gonna what you're planning to do, what the story's going to be. And then
MattGPT says: it says it's you know, you really want one or 2 kind of strong things that you're trying to follow through on the Fred here. Right? So top 10 States
MattGPT says: commute accidents. This is really sort of just weaving together or subseting the data in a little bit. And then you can see.
MattGPT says: you know, reusing bits of code, adapting bits of code, new bits of code there. like, that's yeah.
MattGPT says: not not saying these all perfect and great, because we don't have state
MattGPT says: it's across the bottom here. So it's not clear exactly what's going on.
MattGPT says: and then, you know, looking at some correlations, bringing it all together.
MattGPT says: so I don't know how many words or how much code is in there. But sort of like. that's a reasonable sort of lamp. Here's another one.
MattGPT says:  this one! I could do a little bit more framing, I think right and a little bit what's going on. But still you get the idea right? How works, how they bring the data together?
MattGPT says:  And then
MattGPT says: a few graphs, making some observations on that and then bring it together. So I don't know again. I don't know exactly how many
MattGPT says: words is in that one traffic fatalities.  so some, you know, some people take an angle or have taken an angle where they're explaining more sort of like the code in a step by step detail.
MattGPT says: I want to see a little bit more practice of, you know, showing sort of like story, like stuff. But still you're into. You know, you're having the code sales into leave that they decide to put these 4 plots into one. Yeah, and then
MattGPT says: got sort of like. a nice job of explaining what's going on. Want to see there. and
MattGPT says: and so on.
MattGPT says: see?
MattGPT says: I think that's supposed to be unintentional side effects.
MattGPT says: yeah. So
MattGPT says: yeah, that one could do this would definitely is
MattGPT says: on the mockdown. And the explanation sort of thing.
MattGPT says: how does commuting impact traffic.
MattGPT says: Rough idea. What I'm looking for is
MattGPT says: Nu don't just so say, we looked at free data sets in this assignment. First, we did this first. We do this, but you know, because we've already done that right. You've done the question. So now it's your you know. The whole point of the
MattGPT says: the questions are to get you familiar with the data sets so that you can practice doing that stuff. But in doing that you should be thinking like, oh, this is kinda interesting. I wonder if this, you know. So that's a really good. If you come up with a question that could, you could draw a possible free data set
MattGPT says: to ask a question. That might be a you know, that could be your angle and then set that up and walk your readers through it.
MattGPT says: You don't have to explain all of the code in detail, which is like I'm using pandas and the Pandas group by and stuff like that. But like, what if you put a block of code in? Say, you know, first, we have to load the data in, subset it for the years that we're interested in
MattGPT says: code. And then we can now ask the question, and you know what is, what are the top 5 states that
MattGPT says: in terms of traffic fatalities
MattGPT says: code. So here's what I'm seeing and then. Now we can bring that together with the top 5 States, or you know all the where these States come in the commuting, or where these States come in the
MattGPT says: physical activity metric, and so on.
MattGPT says: Okay? And so then just walk your readers through that, and then, if you do include a figure
MattGPT says: like this, then you really need a mark down cell right after it, where you kind of like say, so. What we see here is just, you know, each pair of bars represents this, and the pattern I'm seeing there. That we can see really clearly is X,
MattGPT says: okay? And then you should. You know all of your you're walking, your read, approve, and then you get to the end and conclude it and say, Well, yeah.
MattGPT says: you know, obviously, there's any limited conclusions that we can draw, maybe, or maybe you have a really clear conclusion, or turns out that there isn't a strong association between X and Y,
MattGPT says: and so on. Yeah.
MattGPT says: So yeah, it's just, you know. Don't you know
MattGPT says: there's only a limited lamp to them? But yeah, the the thing is to practice your
MattGPT says: pulling out. You know you you. The analysis has got you your understanding of the data, and you should have some patents. And now it's doing the storytelling part right? Which is, then how do I weave this altogether. But it's a data.
MattGPT says: It's a, you know, it's a data blog. So you want to actually show the appropriate bits of code and
MattGPT says: data tables that come out and plots that you do to to
MattGPT says: at each stage, and, you know, sort of narratively.
MattGPT says: yeah, just so, yeah, Umhm. So we just so Max asked the question about that. But yeah. So we hadn't actually
MattGPT says: use this one before. But what you can do is add in
MattGPT says: to what you've got. There is.is in
MattGPT says: dot, is IN? And then in parentheses.
MattGPT says: Put your years to plot.
MattGPT says: Okay? And then close parentheses and then close square brackets.
MattGPT says: So what that says is, do a test on the year value is it in this list the list says, 2011, 2016, 2019. So that's saying
MattGPT says: 2,011, or 2,016 or 2,019. Okay, so that that filtered Ds should now have those
MattGPT says: things that. So if you do a
MattGPT says: if you look at that.
MattGPT says: Okay? Then, yeah, looks like it's got those right? So then what you can do to plot it would be,
MattGPT says: what do we remind me what the plot looks like. Oh, yeah, it's by region and by year. Okay, so you wanna do. Probably easiest thing to do is do a group dot group by right filtered df, dot group by
MattGPT says: and then in prem season square brackets. You're gonna put the 2 fields. So I go by region first. So region.
MattGPT says: Yeah, comma year.
MattGPT says: I can do that in the same bracket right yeah, so that what that's saying, is, first, group by the region variable and then group by the year.
MattGPT says: Okay, so you're getting 2 levels of grouping right? And then close that. And then we want to get the commute so close parentheses
MattGPT says: and then select the commute percent. The
MattGPT says: No, you do. So now you want to. Now you want to select the variable that you want to take the mean of.
MattGPT says: okay. So what you've done there is that you've created subsets of the filtered data frame by region and by year. But now you want to select the commute
MattGPT says: PC. Column so square brackets in quotes commute underscore. PC.
MattGPT says: Yep, that's it. And now you can close web brackets and take the mean of that.
MattGPT says: Don't mean.
MattGPT says: okay, run that.
MattGPT says: Okay. So now you see, what you've got is you've got 2 levels of grouping. You've got each region, and then you've got each of your free years, and then
MattGPT says: you've got the mean for that year. So in the far West, in 2,011, and so on. So the last thing to get it to be look to look like a wide table is that do unstack to it.
MattGPT says: Okay? So that's why you would. That's the easiest way to do it is get your order right? So say, region first and then year, and then that's giving you 2 levels of grouping unstack just unstacks it so that the second level becomes that. So then
MattGPT says: if you
MattGPT says: you can do all in one line, if you want. You could add a plot to the end of that plot
MattGPT says: and then plot. Yeah, cause it's the plot function. Okay? And then you need to specify in in the parentheses the kind. And so you want to say, kind equals bar.
MattGPT says: And that will be in quotes
MattGPT says: string. Yeah, that's right. Now that close premises and try that.
MattGPT says: Exactly right? So you see what it does, is it when you have, when you get the table that looks like that one that you just had
MattGPT says: each row. So each row was a region so each row would be given a group, and then each column becomes a bar within that group.
MattGPT says: Okay, so you had free. You had free columns for years. Right. And so what it's doing is it's saying, oh, far West, okay, free bars, 2,011, 2,016, 2,019 next row Great Lakes.
MattGPT says: which columns 2011, 2016, 2019. Okay, so that's the
MattGPT says:  Now, if you just just if if you change bar there to line.
MattGPT says: Just just so you can see what it does right? It will plot it in the same way. But instead of bar charts, you're gonna get free different lines.
MattGPT says: Oh, wow, okay, so it's just like in this case, we're going for a group botch up. But it's just like, once you have the data in the right format, which is like in that table like above, where you've got rows for group, for regions
MattGPT says: and columns for the year. Then you've got what you need. So then you can just add in underneath that line the plot, the PLT. Functions to add a title. and so on, right, so that it looks like the.
MattGPT says: But does that make sense? So so the set the steps in there. But you did a lot in that third line like we did
MattGPT says: we? We did the grouping, and then we did the mean. And then you did the transformation from a long table to a Y table, and then you plotted it right.
MattGPT says: You say you can do that all in one line like that if you want you could break it.
MattGPT says: Yeah, that's right. So so what the all the group dive does is it splits up your data frame into sub
MattGPT says: sub data frames based on the values in the one or more columns that you give the group buying.
MattGPT says: So each state has 3 rows. And so then what group by region says is like, Okay.
MattGPT says: look at the region column and
MattGPT says: group together all of the roads that belong to Far West and then to Great Lakes. And then, you know. So those become your subsets that you work with.
MattGPT says: so that's that's all that group by does. But that's a lot. But that's what it does. In this case, we're saying, actually want the subsets to be a combination of 2 values, the value in the region column and the value in the year. So we want far West
MattGPT says: 2,001,
MattGPT says: all of them, all of the rows that have those 2 values.
MattGPT says: Give me that as a subset. Then I want far West
MattGPT says: 2016. That's a separate subset.
MattGPT says: And then I want far West 2019. That's a separate subset. Right? So that's what group buy is doing. It's splitting up your data frame into a whole series of smaller chunks that then you can do something with each of those chunks. Right?
MattGPT says: And so then the question is, what do you want to do? Well, in this case we want to actually take all
MattGPT says: all of the values in the PC. In the commute PC. Column for each of those chunks and calculate a mean. So we get one mean for each chunk.
MattGPT says: right? So for each
MattGPT says: each region and year, combination far West 2,011. What's the mean of all of the States in
MattGPT says: the far west region in 2011.
MattGPT says: What's the commute? PC, mean? Okay? And that's what that table is. All the unstack is doing is turning it from that one that we saw. That was like
MattGPT says: long form where you had for each region. You had 3 separate values 2011, 2016, 2019.
MattGPT says: If you do unstack, then it flips it around. So instead of getting free rows, you get free columns
MattGPT says: for the year for each year of the 3 years.
MattGPT says: So, you know, it's fine to put it all together like that. But like, make sure you understand what each step is doing, and sometimes it's better just to. So you know, you can always just run each part of it go from the left. So if you took filtered
MattGPT says: df, dot group by and then see? Saw what that did. And then you see what adding the commute underscore PC, dot mean does, and then see what adding the unstacked does, and then
MattGPT says: and so on. You know.
MattGPT says: So I'm happy to hang around for as long as people have questions and stuff. If you don't have any questions and you wanna jump off. You're welcome to do that. But I
MattGPT says: can stay around for as long as people have questions. We can just sort of treat it ongoing office hours. Okay, so.
MattGPT says: but also, if you've got to go, then go But if you do have questions
MattGPT says:  just send me an email. I'll try and respond to the email. Or if you wanna set up a zoom tomorrow.
MattGPT says: We can do that, too. Okay.
MattGPT says: How you doing
MattGPT says: Thanks. I saw your email,
MattGPT says: with the data set. So yeah, I think so you're happy for people to have that data set and play around with it.
MattGPT says: Okay.
MattGPT says:  so yeah, what I think we'll do is after breaking one of the classes, we can do it as a proof exercise, like we did with the
MattGPT says: shooting data set. I think that work pretty well to have people work on it together.
MattGPT says: And
MattGPT says: So what's tell me a bit more about the history of the
MattGPT says: gotcha, yeah.
MattGPT says:  yeah, let's have a quick. I'm just gonna
MattGPT says: okay?
MattGPT says: So you've already kind of spent some time with these data files. Right?
MattGPT says: Just looking at your email again. Hold on.
MattGPT says: huh?
MattGPT says: And there's one called Views. Right?
MattGPT says: download them. A sec.
MattGPT says: Okay? Okay, I don't really got it.
MattGPT says: Okay. there's 11 of you, right?
MattGPT says: Okay. So the monthly views is.
MattGPT says: I see. Okay.
MattGPT says: alright
MattGPT says: factor in those. That
MattGPT says: isn't a data holder to be
MattGPT says: some reason it was blocking me downloading the Csv.
MattGPT says: this is all right.
MattGPT says: But I think I mentioned this to you before, which is like
MattGPT says: you can do.
MattGPT says: Read. excel.
MattGPT says: read an excel, file
MattGPT says: Okay. So this is a good example of why data? Right? Let me sort of thing. What I would do with this, then, is.
MattGPT says:  yeah. Well.
MattGPT says: okay. So then.
MattGPT says: binding name.
MattGPT says: Yeah. So I would like.
MattGPT says: put into a long data frame
MattGPT says: the
MattGPT says: Oh, yeah, we don't want 40 years.
MattGPT says: second.
MattGPT says: there we go.
MattGPT says: Okay?
MattGPT says: And this is what in what was 2.9 in what?
MattGPT says: Thousands?
MattGPT says:  okay. So that would give you just like a year summary
MattGPT says: and
MattGPT says: follow really nicely.
MattGPT says:  interesting. Okay? So when did you say the strike? What?
MattGPT says: yeah, okay.
MattGPT says: yeah. So this is the kind of thing I was saying, it's really
MattGPT says: nice about using plotly, right? So doing exploration, we can see? But you can get the same thing from this right? Well, just just
MattGPT says: we stretch this out a little bit.
MattGPT says: So what's this second peak? Do you think the strike was in
MattGPT says: beginning of 2020?
MattGPT says: Okay, and then this is the article history. What's that?
MattGPT says: okay. So it looks like we do.
MattGPT says: yeah, okay?
MattGPT says: And then
MattGPT says: actually row 2.
MattGPT says: Let me take care of it.
MattGPT says: Yes,
MattGPT says: okay. So that's nice. So then you've got like
MattGPT says: tags.
MattGPT says: Oh, right? So, not not super consistent, necessarily.
MattGPT says: but alright. So tags things can be multiple tags, right? So that tags we can. You can think about like breaking them out into separate
MattGPT says: sort of things.
MattGPT says: pull them out. But this is nice that we've got date specifically here. Right? So you can just do
MattGPT says: the equivalent of what we just did, which was like.
MattGPT says: So this is a string, right? But this and we'll we'll cover this
MattGPT says: soon. It's like
MattGPT says: a date time. Object, right is. It's gonna complain.
MattGPT says: The month years missing.
MattGPT says: Oh, no, we need to give it that
MattGPT says: columns
MattGPT says: great
MattGPT says: and that string. Okay.
MattGPT says: so let's try.
MattGPT says: So what they start what they? I,
MattGPT says: this is working. Let's have a look.
MattGPT says: Let's see.
MattGPT says: Yeah.
MattGPT says: DD 10.
MattGPT says: yeah. Okay.
MattGPT says: So yeah, there we go. Okay.
MattGPT says: so this is, this is pause that string and try to figure out like.
MattGPT says: is this a valid date time thing. And so a date time object has
MattGPT says:  a whole bunch of
MattGPT says: element. So you can get like, yeah. you can get month
MattGPT says: month number you can get. Day should be a day number.
MattGPT says: Yeah. So there's a whole bunch of things that you can do with that which is nice. So it's aware of what dates are right? And so you can get like week of the year.
MattGPT says: and stuff like that. So you get like one to 52.
MattGPT says:  So what's nice about this is then
MattGPT says:  like, if we just
MattGPT says: set the index to be that field
MattGPT says: like that and then we could just
MattGPT says: Do this
MattGPT says: group by  yeah.
MattGPT says: Resemble
MattGPT says: down to
MattGPT says: okay. YM.
MattGPT says: okay.
MattGPT says: weekly.
MattGPT says: Okay, so what this is done is taken. It's resampled down to the week level. If we resample to the month level
MattGPT says: and it's grouping all to, you know, is it's using the date time thing and saying, like everything that belongs to the same month, into the same year.
MattGPT says: Put those together, and then we can get the size of that, which is the number. So this shows you over time at the month year.
MattGPT says: Yeah, the month at the month level across all the years. How many articles were published?
MattGPT says: Oh, very cool, right? And so. you know, and if we you can play around with that in different sorts of ways, right? So what I'm thinking is that then you can.
MattGPT says: This is views
MattGPT says: and trying to think, what do we have that level at
MattGPT says: use?
MattGPT says:  thank you.
MattGPT says: And then we'd have to say English.
MattGPT says: Some people
MattGPT says: this could be the same as actually what we had there.
MattGPT says: But yeah, these are then, I think, be on the same scale.
MattGPT says: So the question is how to kind of align them up, but basically
MattGPT says:  what's kind of interesting here, it wasn't. That was suddenly a lot more articles on average during that strike period. Right?
MattGPT says: so
MattGPT says: yeah, so that would be the thing to try and start to make sense of this, and that's going to be
MattGPT says:  This is going to be some pattern over the year. Right? So if you look at the
MattGPT says: Alright. So if you so again, this is what we can do with our history. Yeah.
MattGPT says: 18.
MattGPT says: And then, if we said. Give me.
MattGPT says: Is it like hanging right now? So maybe
MattGPT says: we come here
MattGPT says: right? So that's
MattGPT says: that's the pattern which makes sense right of Sebastian.
MattGPT says: But you have so first week of the year through to like the twentieth or nineteenth, and then you have
MattGPT says: very little going on in summer, and then you have a full patent. So that's across all.
MattGPT says: right?
MattGPT says: So you do. Yeah, that's the kind of thing I'm thinking that we can play. It's quite nice data
MattGPT says: in that way for playing around with that.
MattGPT says: What else do we have offer
MattGPT says: title
MattGPT says: now? Title? We'd have to get into some sort of like
MattGPT says: we did want to do something with title.
MattGPT says:  do some sort of
MattGPT says: text processing on it to see if there are like
MattGPT says: certain keywords in there, I guess.
MattGPT says: yeah. And then you know the tags we can. You can think about how, what might be the meaning of the tags? Right suddenly you can pick out anything that has features in it, or there anything that has opinion in it like frequency of opinion pieces.
MattGPT says:  so, yeah, this is, this is, I think there's some stuff in there, like.
MattGPT says: let's see.
MattGPT says: interesting.
MattGPT says: yeah. And then it's just like, if anything is figuring out like
MattGPT says: this makes sense to the strike like what was happening here. These are covid years, too.  and
MattGPT says: sort of like in this period here, too, right?
MattGPT says: Usually like, what's the is it daily like? Is there a schedule for publishing? No, it's it's all online. So it just goes up when it's ready.
MattGPT says: Right?
MattGPT says:  yeah. So that's why I guess we thought.
MattGPT says:  it's gonna be way too noisy. But like that's actually at the day level.
MattGPT says: right? So that just shows you kind of like.
MattGPT says: And
MattGPT says: but then I guess you could also do something like
MattGPT says: day of the week.
MattGPT says: Oh.
MattGPT says:  we just got there.
MattGPT says: And
MattGPT says: you know, you're good at and and I'm really, always really pleased when you interact with the class, right? So if you can call upon you to maybe give it some context, right? Then we can do it as a as an exercise, and that's, you know, some limits to it. But it's like, it's
MattGPT says: it's interesting. That's enough. There, I think, to try and sort of see what stories we can pull out of it.
MattGPT says: So yeah, thanks. Thanks for doing that.
MattGPT says: And yeah, maybe we can check in.
MattGPT says: we can, we can see
MattGPT says: alright? Sounds good. Alright, thanks. Yeah. Yeah.
MattGPT says: I'm
MattGPT says: good luck today.
MattGPT says: Okay, doing okay? Well, today was continue having to work on
MattGPT says: looking at the
MattGPT says: of the data set that you downloaded anywhere from.
MattGPT says: That's just a municipal equation and stuff. So
MattGPT says: that's a few of us do that quite good. Alright.
MattGPT says: what people are working on.
MattGPT says: So that's the main thing I want to do. There's a few data tests.
MattGPT says: but I've been playing around with have some things that I can show you as well. I mean. But
MattGPT says: yeah.
MattGPT says: you might, if we go over how to like the syntax to site within the short paper Markdown. So one more time. Yes, so
MattGPT says:  that.
MattGPT says: So here like.
MattGPT says: have
MattGPT says: the
MattGPT says: well like. So to actually put one in right. So you've got the you got this working now right so that you can. You can put that in. So if you just do an insert at the point in the Markdown cell
MattGPT says: that you want to do
MattGPT says: He lets me
MattGPT says: studies
MattGPT says: have shown that. Yeah, so I see
MattGPT says:  And so then it's at that point where you'd
MattGPT says: do the insert right here right.
MattGPT says: and then
MattGPT says: and in that
MattGPT says:  And so
MattGPT says: this the tag gets inserted there.  the keypod is actually decided. identified.
MattGPT says: identify the item in those departments.
MattGPT says:  that on the virus.
MattGPT says:  and then it will automatically
MattGPT says: depending on the style that you've set.
MattGPT says: Give you the content of the tag. Right? So that's the thing that
MattGPT says:  it comes out there. So say we want to put in like multiple sources. Then you would just kind of save
MattGPT says: this study. And also and
MattGPT says: my problem is that there's no drop-down options when I do that like here. Yeah.
MattGPT says: I connected via. or I imported my into 40, I'm pretty sure I important it right, but if there's no drop down
MattGPT says: for this like, for example.
MattGPT says:  I'm not sure how to use
MattGPT says: like that moving up analysis on whether. Yeah.
MattGPT says: Okay? Alright. So let's do refresh on the whole thing.
MattGPT says: You got to do.
MattGPT says: Good evening.
MattGPT says: I don't think it's connected. Okay, so let's go to the setting
MattGPT says: Yeah, you can. Just
MattGPT says: you have to read it. Then, if I can't believe that
MattGPT says: alright. So then, under that one, yeah.
MattGPT says: no. I had. They generated a key for me, and I believe that I don't know why this is the key on me.
MattGPT says: I guess I get generally enough.
MattGPT says: That's right. No, not good settings.
MattGPT says: think that's what's happening. I was
MattGPT says: That's all
MattGPT says: navigate way. But
MattGPT says: Anyone else having issues using that getting the citations to come in.
MattGPT says: You fix your, you're the still working man. Right? Yeah.
MattGPT says:  yeah. So what we're doing. Like, if you want to be working on your paper. You can do that, and I'm happy for you to use, you know. and if you have questions, or problems for stuff. We can fix that but also we started
MattGPT says: looking at
MattGPT says:  using the template.
MattGPT says: the open data example. Okay, so everyone should have selected downloaded Csv
MattGPT says: upload it into the data folder. And then
MattGPT says: you can use this these steps as a way to get started. I'm looking at that. Okay? And so,
MattGPT says: yeah, if you if you wanna, you want to do that stuff.
MattGPT says: And
MattGPT says: okay, so a little less structured than we've done before. But I, you know, this is kind of now, I feel like, everyone's
MattGPT says: got to the state where we're ready to start like working on your own data and doing the analysis. Okay? So we've got this kind of formulaic set of steps that we go through
MattGPT says: first stage, download the data, upload it to the data folder
MattGPT says: and then
MattGPT says: load it in, investigate its shape.
MattGPT says: take a peek at it, use the head tail, take a sample.
MattGPT says: What are the rows? What do they mean? What are the columns?
MattGPT says:  and what's the range of the data that's there?
MattGPT says:  what are the columns which are useful? What are the columns that
MattGPT says:  Not mutual questions that you might have.
MattGPT says: Then coming down understanding the data, take a bit noting things.
MattGPT says:  then. Wanna do that. Okay, what are the main
MattGPT says: dimensions that can be used. and I'll be president.
MattGPT says: So it could be that you've got some data that spans over time. Okay, so you've got a temple dimension on that variation by time.
MattGPT says: Could be that your data is
MattGPT says: got some sort of location in and necessarily like geolocation. It could just be like, this is a Zip code, or this is a state
MattGPT says: or some area that again gives you another location to look at.
MattGPT says: and so on. Right? And what demographic variables. Have you got?
MattGPT says:  so identify those  And then
MattGPT says: out of the key column for that, and then you want to go through it to do the summary so kind of horrible one. Then you value account, find out the distribution
MattGPT says:  tricky part. Sometimes, if I could get it going with extra heat or load the data in, and it would be like, you get stuck on that. Let me know if we're gonna help you and see if you can get through all of these steps
MattGPT says: cool. And so I don't.
MattGPT says:  okay.
MattGPT says: it's a huge. It's a lot.
MattGPT says: Yeah. Oh, okay, yeah. Zipzer? Yeah.
MattGPT says:  So what what is the
MattGPT says: there's a metric probably about right?
MattGPT says: Is this? Is it the one with the rain flying?
MattGPT says: We grow? It's like some metric in some combination of
MattGPT says:  yeah. Okay.
MattGPT says: right? So yeah, he grows inactive. A.
MattGPT says: So it's a metric
MattGPT says: in a year, you know, location for a specific demographic.
MattGPT says: I mean by
MattGPT says: I just think this one now, I guess, to do things like endless right?
MattGPT says: So you could say like.
MattGPT says: request that
MattGPT says:  like they wanna do a lot of time, I guess
MattGPT says: and that
MattGPT says: you have to try like roofing by.
MattGPT says: And then you've got year as well. Right? Yeah.
MattGPT says: take take a year, and that's the name you group by, and then
MattGPT says: a list. But give us. And then that's the second one.
MattGPT says: Wherever I get
MattGPT says: grouped by 2 things, and we've seen a list in Budget
MattGPT says: great. And then after, like the metric value.
MattGPT says: Yes, that's the that's your only
MattGPT says: will be married. That would be
MattGPT says: yeah. But it's not like.
MattGPT says: okay. And then you're grouping by what I say. But it's going to meet from
MattGPT says: and locate
MattGPT says: on that.
MattGPT says: And what does the metric value? Well, that's the penis.
MattGPT says: It's actually diesel pairs. So this is like sensitive 6. So the metric name total of the metric is
MattGPT says: right.
MattGPT says: Metrics, particular things like this
MattGPT says: like, did, did you do a value? Count on the bank's name?
MattGPT says: Okay? So that shows you like how many. This is kind of weird. They don't put it.
MattGPT says: I tried to do sat scores. But yeah, maybe you want me
MattGPT says: a
MattGPT says: smoking, so you define it. Filter on that. But
MattGPT says: alright, yeah.
MattGPT says: no. Okay. Because those are the columns. The way you need to do that is filter the road.
MattGPT says: Make it so good.
MattGPT says: Right? Yeah.
MattGPT says: Well, no, you don't need to worry about that on that piece. But now you put it in a book.
MattGPT says: So let's do smoking.
MattGPT says: Okay? And then it's gonna be the data range.
MattGPT says: And then you're gonna select phone like this is metric name.
MattGPT says: Okay. So that's going to give you. Now, you see, in the next line underneath. You put it in the same socket.
MattGPT says: but
MattGPT says: Smoking might as well be up, or something
MattGPT says: equal.
MattGPT says: and then subset the data frame. So you put the name of the database.
MattGPT says: Yeah.
MattGPT says: integrated.
MattGPT says: Okay, whatever.
MattGPT says: So then, now.
MattGPT says: so you see what I'm showing you. But and it doesn't break down smoking by
MattGPT says: my email. Yeah. So for that, you don't need that right?
MattGPT says: What we have done. Yeah.
MattGPT says: So go back to the smoking month effect. So here, yeah, the new block
MattGPT says: like, and service that put in
MattGPT says: when it deals with these.
MattGPT says: yeah, so be nice. Okay.
MattGPT says: you're not up.
MattGPT says: Let me.
MattGPT says: Okay. So you see an out of that. Then breaking that down. So go ahead.
MattGPT says: One way to do it is built on a set
MattGPT says: alone. White label. Right?
MattGPT says: That it funds that level.
MattGPT says: Sorry. That was the correct.
MattGPT says: Yeah. But if you just wanted to do stuff like that.
MattGPT says: the other thing that you can do is say, well. but to to
MattGPT says: for acting to barriers in mind. So
MattGPT says: location and your time and leave your corporate money.
MattGPT says: yeah, so for that, please, if you
MattGPT says: and name that whole thing.
MattGPT says: Yeah.
MattGPT says: Then, yeah.
MattGPT says: not that kind of time.
MattGPT says: What else do you question on that?
MattGPT says: Here? It's web bracket.
MattGPT says: Yeah.
MattGPT says: it's actually a number, but it doesn't.
MattGPT says: aye
MattGPT says: let me put on.
MattGPT says: And this is the metric. That's the magic value. Right?
MattGPT says: Okay. So you can go back and do that. The other thing that you could do is just say, okay.
MattGPT says: you would take out them
MattGPT says: in your reply. You can take out gear but then you can get multiple rows. So you have to go.
MattGPT says: Other than that. There's no level right to get rid of that
MattGPT says: and that. But I don't kid you get rid of that yeah
MattGPT says: That's the mean across all of the years. Probably right? There's an overblown point. You want to get another.
MattGPT says: That's what you know. Once you identify the domain, they might have time
MattGPT says: variable when you got.
MattGPT says: Then I'm gonna think about, do I want to change at the time? Or do I want to look at the change? But or find sometimes
MattGPT says: okay. Vibrancy and immortality that can end up in Brendan would be like.
MattGPT says:  is there a difference between?
MattGPT says: It looks like seconding the data
MattGPT says: plan. You can include sectors if you can go
MattGPT says: then you could, so that I could go to the next time. But yeah, I would.
MattGPT says: Hello.
MattGPT says: feel proud and feel so. All the roads are having mortality.
MattGPT says: Take that into the data frame
MattGPT says: and group by year, and that  and
MattGPT says: then there's a necessary and then see it on that opportunity.
MattGPT says: and then you could take out the air and just get the first point
MattGPT says: difference in my own female.
MattGPT says: And yeah, so
MattGPT says: Then we
MattGPT says: name and then just group it by
MattGPT says: and then get the mean of a message.
MattGPT says: doesn't it? Doesn't break down.
MattGPT says: So which method is not balanced. So, okay, what's the what's the accurate one there?
MattGPT says: Yes.
MattGPT says: I like in second view. But at that time
MattGPT says: there's a little time for that.
MattGPT says: Okay, alright.
MattGPT says: So yeah, I mean, that follows, I guess what happens.
MattGPT says: you know, anybody that I can see time.
MattGPT says: Yeah, I mean that's not really gonna show you anything about that. But what about any group within? Also part of the region? I'd like to deal with your name right in the previous line. So I have the full effect in that.
MattGPT says: oh, no, it's got a
MattGPT says: there we go. Okay? So now, what that's showing you that you can see like, is there any difference
MattGPT says: if we have these unselective pregnancies by
MattGPT says: it's in a low income area, but
MattGPT says: area, which is right by a chemical.
MattGPT says: So you can say, well, actually, then, the mail emailing doesn't matter. Take that out.
MattGPT says: So you're kind of playing around with like.
MattGPT says: and then sort of instead of jobs, you know, with the year, really?
MattGPT says: Well, that's good.
MattGPT says: So I mean, there's a lot of what? What the background of all those.
MattGPT says: So I'll go back to the
MattGPT says: I don't know.
MattGPT says: Yeah.
MattGPT says: So it's the
MattGPT says: bibliography. I think that's the wrong.
MattGPT says: yeah. I think that if you, I think you're invited to be able to do that.
MattGPT says: oh, so I would just delete all of that. Yeah. Oh, so it's just like, yeah.
MattGPT says: this wagon is filling the hotel right? Because the other hand.
MattGPT says: so you can try and bring in a site again. I'll get changes to the long now.
MattGPT says: Okay, great. Now, if you do want to go with me, then what you do is like tennis in the market, though.
MattGPT says: Okay? And then but then let's put it in the title for them. I struggled.
MattGPT says: then leave it and say, and then, yeah, and then leap off.
MattGPT says: So I can say, Hey, this is weird. I'm not worried.
MattGPT says: And each of that's the only guess. Is there.
MattGPT says: we were trying to renew. Simon was just.
MattGPT says: is there? So it's only doing on that. It's not doing on the breakfast.
MattGPT says: Yeah, this is going on. The one thing you add in responsible
MattGPT says: and
MattGPT says:  For now I would say, don't worry about putting
MattGPT says: right when I ask.
MattGPT says: yeah, let's try doing that.
MattGPT says: So. Yeah.
MattGPT says: make sure you want. Yeah.
MattGPT says: And we need to. Actually. there's all of that. Actually.
MattGPT says: Yeah.
MattGPT says: I have one of these
MattGPT says: like. And then it started flapping. And then, it was. It was the worst thing that they
MattGPT says: more. But you know, possibly. yeah, that's what I did.
MattGPT says: That is funny. Because I mean, finally.
MattGPT says: Oh, there you go.
MattGPT says: no, it's frustrating writing.
MattGPT says: And I'm guessing. Wow.
MattGPT says: Yeah.
MattGPT says: whenever I see it's gonna
MattGPT says: right?
MattGPT says: okay.
MattGPT says: okay, yeah. So I'm
MattGPT says:  do you have any. Again.
MattGPT says: we make franchise. Yeah.
MattGPT says: Help me out.
MattGPT says: Go ahead. And
MattGPT says:  okay, that's not blaming.
MattGPT says: Okay, okay.
MattGPT says: I am certain that
MattGPT says: yeah, yeah, great.
MattGPT says: Hmm.
MattGPT says: anyone asking specific questions or problems. Yeah.
MattGPT says: got the data. And I'm just kind of like a calling. Yeah. And so I took out like.
MattGPT says: I wanted to sound out like
MattGPT says: male call. But I wonder how the data?
MattGPT says: Yeah. So there's 2 ways you can do, you can do it. Look. And
MattGPT says: you can just create a list because you want to keep.
MattGPT says: And then let's do that. But like there's more. Then that way. News drop.
MattGPT says: There is an angle drop.
MattGPT says: aye. also trade a friend that I put in here. See? This being better thing.
MattGPT says: Okay? So
MattGPT says: nothing, so you can do.
MattGPT says: and then give it columns.
MattGPT says: and then you give it a list of the ones that you don't want. So say we wanted to get rid of the geom.
MattGPT says: What that meant.
MattGPT says: Okay? And we wanted to
MattGPT says: let me.
MattGPT says: Okay. So you see how that works.
MattGPT says: So from your day with rain, I'll drop the name.
MattGPT says: Hold on to give it to this one. You won't drop now, that's just going to display that. So then, if you wanted to. And then what you would do is name the result? Right? So call this like, Eb, 2. Yeah.
MattGPT says: that's probably the easiest way to draw specific comments by.
MattGPT says: And we're we are new. And but like.
MattGPT says: I realized, we couldn't use all of the same cultural zones that I had just
MattGPT says: So yeah, that way. We still bring public right? So I mean.
MattGPT says: yeah. So you remember how we set up a building like based on. On.
MattGPT says: But there's 2 ways of doing it like
MattGPT says: again. And
MattGPT says: right. These are.
MattGPT says: remember, each row has a index. This first one is the index
MattGPT says: by the focus of America. Right?
MattGPT says: So you can.
MattGPT says: you can select columns using
MattGPT says: locate. Right? So
MattGPT says: I'm just saying 0
MattGPT says: and that will give us 0. It can give it a range like 0 check.
MattGPT says: Right? So if you wanted to try and select specific columns like that.
MattGPT says: okay, so that's using the index. Now, the other thing that we haven't covered like.
MattGPT says: And this is a bad.
MattGPT says: So example for this. But let's say you can actually reset the index or set the index to be one of the columns. So in this case, if I said.
MattGPT says: use the object Id instead of the numeric number, as the index.
MattGPT says: what that does is that takes that column.
MattGPT says: and it puts it in the
MattGPT says: it becomes in that.
MattGPT says: And so why is that useful? It's because then you can select things by
MattGPT says: the index, right? So we could just say.
MattGPT says: let's get that particular
MattGPT says: item out there. Okay, so that's that's one way to do it. But the way that we kind of focus on with more. This idea of filters, right? So that we could say, like.
MattGPT says:  let's say that we want.
MattGPT says: yeah. So we define a filter where I would say
MattGPT says: data frame, and then the column or columns that we want to filter by. So this data is not very meaningful. I mean, it doesn't have a lot going on in it, so. But I can use the object. Id.
MattGPT says: and if I want all of the object ids, because it's a number
MattGPT says: less than 60.
MattGPT says: Right? Then, that filter. Then I applied to that data frame.
MattGPT says: Okay? So now that
MattGPT says:  okay.
MattGPT says: what we got back.
MattGPT says: Meet it on me
MattGPT says: on the one without this chunk
MattGPT says: There we go.
MattGPT says: Okay, so you can see, like all of the rows that I get back just have a value which is less than 60
MattGPT says: is. So you have to think about. How do I set up my filter like. So
MattGPT says: if it's numeric, it's easy. Use ranges.
MattGPT says:  If you wanted to have, you could also select specific ones.
MattGPT says: you can say is in right, and then you can give it a list. Right? So say, I just want to column 45, row 45, and
MattGPT says: 1, 13 or something like that.
MattGPT says: And this is specific to the or whatever. But
MattGPT says: that filter, then, will just pick up those rows
MattGPT says: oops. That is end
MattGPT says: Okay? And so then the only rows that I have left will be those ones like that.
MattGPT says: Okay? And the nice thing about filters is, once you define a filter, you can just invert it. So if
MattGPT says: if you're trying to drop specific rows, and it's easier to think about like just the rows that you don't want as opposed to all the rows that you do want.
MattGPT says: then define it, as you know, you can flip that round. So I'm saying I don't want. I want all of the rows apart from the ones that have the object Ids listed in here.
MattGPT says: Okay? And then you'll see that I get. It's just drop those specific ones.
MattGPT says: Okay, great. Remind me of what you were doing.
MattGPT says: Oh, yes.
MattGPT says: you know. I mean, it's yeah, I mean thinking about like.
MattGPT says: but have that
MattGPT says:  But
MattGPT says: you know, it's like
MattGPT says: that. But the the algorithms itself, that's the data itself. Right? So the issue with Gen. AI, because it's
MattGPT says: at the moment, primarily, that's related or image related. Right? It's like
MattGPT says: based on it. the state of the topic.
MattGPT says: So I don't know if you can say that, you know, if you go to Chat Gp. Or something like that.
MattGPT says: Come in, come in a story about a doctor, you say today?
MattGPT says: Alright, and
MattGPT says: I mean
MattGPT says: let it generate that, and you do it again and there again to save you the bucket. People have done this like 95% of time. It gives you a male name for a doctor.
MattGPT says: Right?
MattGPT says: 95% fine. Get on it.
MattGPT says: And then you could. If you want to do intersectionality, then it's like in the things that this includes
MattGPT says: female. The nurses tend to be. But then they have a range of things difficult to tell their from the name. You know what I mean. So there's not all of that kind of
MattGPT says: where does that come from whether like, that's just
MattGPT says: what is out there. And the data that's been trained on seem to represent our biases. But also what goes on with the inequalities.
MattGPT says: Maybe
MattGPT says: decision algorithms more like, you know, go to the bank.
MattGPT says: not the level
MattGPT says: they take. All these demographics like your age to then know, like, what role does.
MattGPT says: How are those campaigns and knowledge?
MattGPT says: So you know, that was just kind of like.
MattGPT says: yeah. So you don't necessarily have time dinner. But you don't want to write about that plan.
MattGPT says: Think about what it is.
MattGPT says: Specifically, we're in that full topic.
MattGPT says: yeah, you're right.
MattGPT says: Thank you so much.
MattGPT says: You go to. Yeah, okay.
MattGPT says: And then I also love touching feedback. My.
MattGPT says: oh, alright, I gotta have the.
MattGPT says: And then
MattGPT says: I wanna open this one.
MattGPT says: Today. We're in the same.
MattGPT says: So information.
MattGPT says: what if he broke at them?
MattGPT says: in the city?
MattGPT says: So in that sense. But buying is not useful.
MattGPT says: variable. Right?
MattGPT says: that point.
MattGPT says: Where did you find this from everybody.
MattGPT says: everything time with good intent, fine
MattGPT says: interacting scientists? That's what they tend to be.
MattGPT says: It's not very interesting, but they've been. They've been even satisfied, or
MattGPT says: theme so that they work for a function right?
MattGPT says: Be honest, a lot of.
MattGPT says: But it's an interesting data plan.
MattGPT says: So
MattGPT says: yeah, there's no room unless they say, like where they got the data from. And then you can go and try and speak for yourself or download it.
MattGPT says:  thanks.
MattGPT says: yeah. I mean happy. I will go by that. Thank you. Privacy and
MattGPT says: I can kind of see.
MattGPT says: But yeah, that's equilibrium
MattGPT says: depends what works like.
MattGPT says: Okay?
MattGPT says: And then we'll start with this. Is it easy on our planning point.
MattGPT says: What else is doing? That
MattGPT says: So, whatever variation is in there, I mean a bit different.
MattGPT says: But
MattGPT says: cause you can see what? What can you do with it? Right in terms of
MattGPT says: the ratings are used, for example.
MattGPT says: okay.
MattGPT says: So why don't you try and still do that like, how many products are there like? And then
MattGPT says: county road is the same possibility.
MattGPT says: How many products are there in there?
MattGPT says: And then.
MattGPT says: the one in front
MattGPT says: continue to create a ranked table of products by ratings, by meet ratings.
MattGPT says: Do me that
MattGPT says: You want to ring right?
MattGPT says: is that.
MattGPT says: And I'm also product for that. There must be like, did you do a value? Count? Yeah. yeah, it's like a long title. Right?
MattGPT says: Okay? So that's going in with product. But
MattGPT says: the highest rated. So you put that
MattGPT says: I would look for these.
MattGPT says: Alright. Yeah, right? Is that right?
MattGPT says:  there anything else about category?
MattGPT says: a pretty limited site.
MattGPT says: And that's trouble back to.
MattGPT says: So I mean, you know, you can't read it there.
MattGPT says: basically. And data as well understand data
MattGPT says: decomposable, useless default, little things.
MattGPT says: There's no time dimension. But there is a product dimension.
MattGPT says: and there it
MattGPT says: categorical. And there is a rating.
MattGPT says: tolerating for kind of things. We'll kind of, do
MattGPT says: yeah, the problem
MattGPT says: so many minutes.
MattGPT says: their room in their room.
MattGPT says: Yeah, you can try something
MattGPT says: about what to do with data set.
MattGPT says:  is there, how they are always different data
MattGPT says: Those data sets are hard to get.  But
MattGPT says: if you go to the same product on Amazon
MattGPT says: that
MattGPT says: interested in the marketing product.
MattGPT says: buying them in data set. Now a lot of data that's time before.
MattGPT says: So I'm generating my next question. I'm doing a little bit like this.
MattGPT says: But it is, I would say, like. try and do the right stuff.
MattGPT says: stay there.
MattGPT says: So vote just take a look
MattGPT says: or signing the following, that editorial
MattGPT says: yeah. I mean spreading from
MattGPT says: modules.
MattGPT says: You
MattGPT says: Yeah. So we'll cover that.
MattGPT says: depends on the site. Right? Simple.
MattGPT says: What would be 5 min. That was 8, and the backpack really is, you know.
MattGPT says: And then you look at it and say, like, what the where's the data that I want pull out. Located
MattGPT says:  island
MattGPT says: browser.
MattGPT says: And
MattGPT says: I was doing this manually where I go to the box, and I type in
MattGPT says: free thread
MattGPT says: right? And then so
MattGPT says: and then the results would come up, and then I'd be like, Oh, yeah, I want to have a little bit.
MattGPT says: So you
MattGPT says: you have to you have to give it right? So that
MattGPT says: how would I get it?
MattGPT says: And so once they go to the home page with the third box, and they're sparing
MattGPT says: bunch up. and then we
MattGPT says: they're on the bottom of the page. There's a list of results.
MattGPT says: Get the body want to breathe, or whatever anything and more, and do
MattGPT says: want to understand. But then, navigating this, doesn't many times.
MattGPT says: Yeah, then then.
MattGPT says: and then extract the data and then put it on.
MattGPT says: But they said, Yeah.
MattGPT says: and like to say, there's a way that we.
MattGPT says: So you have to do something fine.
MattGPT says: but always the threat.
MattGPT says: We don't want them to do it by. and how to take it in a policy.
MattGPT says: and I'll note that what you do for the bike
MattGPT says: and
MattGPT says: and then
MattGPT says: so you can still get along.
MattGPT says: So yeah, I mean, that's my fault. Yeah, we'll wait yeah
MattGPT says: an hour. But see what you can get on that thing.
MattGPT says: did you manage to fix that if you I managed to get the
MattGPT says: the assignment for feedback? But for some reason
MattGPT says: still going up
MattGPT says: like when I fetch me that
MattGPT says: Thank you, and
MattGPT says: let me copy out of the thank you. I mean. hmm!
MattGPT says: It's a mystery to me why it does. It's kinda glitchy. So
MattGPT says: okay, Shiva.
MattGPT says: hold it for that. Yeah, I should have it in there.
MattGPT says: I deleted the feedback folder. So don't do the click feedback again. And that just isn't working right. So just go to the folder
MattGPT says: and then you'll see the inside annotated bibliography updated.
MattGPT says: There's a feedback folder, and then inside that there's the.
MattGPT says: and there'll be assignment. That should be where the feedback.
MattGPT says: Alright. Thank you.
MattGPT says: I was just checking to see the action, to give you some feedback. But I did. Yeah, I think it's a really interesting topic. really, financial paper. So
MattGPT says: think about like focusing on some of the
MattGPT says: the issues are one of the ones I was wondering is like
MattGPT says: a
MattGPT says: You know this. We have this general sort of like
MattGPT says: elevation of an artist and working in cobra, and maybe what people have.
MattGPT says: But so then, you know, it's like, if you work in an office, and then the day I comes in and does your job with generating emails or writing memos, or something like that, then it's sort of like, oh, well, you know, it's live.
MattGPT says: But if you're a creative. How can you write writing scripts. screen or film, or whatever? Then suddenly, it's kind of like, you know.
MattGPT says: the robots are coming. Yeah, or AI or something, completely replace screenwriters, and they got into a very big
MattGPT says: section about what creativity is, and like the the artistry of creativity in the way, that I haven't actually seen quite a lot of the discussions around right about it. Right? It's very sort of like provided value.
MattGPT says: actually, what is feared, if any.
MattGPT says: and in some ways I would argue that actually well.
MattGPT says: Yet what Jennifer AI is doing
MattGPT says: is actually creative in the in a very specific sort of understanding of that right
MattGPT says: numb trying to be years ago in linguistics about creativity in language, right? And the idea that like
MattGPT says: there's the basic building blocks and language, but like no 2 sentences that have ever been said are the same. Right? I think it's probably not quite right on that, but on on the whole, it's like, Yeah, this is
MattGPT says: small set of building blocks, and then you have. It generates an infinite number of different
MattGPT says: sentences. So
MattGPT says: so he called that like creativity of the system, right? And so.
MattGPT says: if that was just general.
MattGPT says: once upon a time, there was a right, because it's in the context of very.
MattGPT says: And so why this technology is so amazing. And
MattGPT says: what it produces is that
MattGPT says: seems to pick up all of those contextual associations.
MattGPT says: And so what we might consider like pregnant.
MattGPT says: So I can play around with this. You can sort of ask your question and say like, Give me the answer in the style of playing gangs the rap.
MattGPT says: but it used to be much more fun to do that, because, like it was.
MattGPT says: he's us really nice.
MattGPT says: and I don't.
MattGPT says: But like. you know, so
MattGPT says: it's hard to tell exactly what they are. But but at basic level has this ability to to generate. Really, Pdma.
MattGPT says: it's a hybrid processor. Yeah, it's kind of like a, you know building. And but it's really amazing in that. And so
MattGPT says: but
MattGPT says: so you know, where does creativity write? If you're writing a film script right? You know.
MattGPT says: clock development.
MattGPT says: it's gonna you know, it's all it's doing is really is giving you back what is
MattGPT says: seen before. But it's learning
MattGPT says: all those little proteins.
MattGPT says: The
MattGPT says: yeah, that's also pretty interesting. But anyway, yeah, I think it's really interesting. Button, you know you want to try and see if we can pick out one or 2 things. Yeah, and a little bit there.
MattGPT says: hang out with that.
MattGPT says: you guys doing? I'm not sure why those have true values here.
MattGPT says: I think it's
MattGPT says: oh. that means for opening.
MattGPT says: Yeah. So that's so like
MattGPT says: that error. Yeah, I realize they're like true
MattGPT says: Oh, I see here. So you put like into 3 things that. So yeah, this means that not a number right? Then, that's gonna be.
MattGPT says: I didn't have any.
MattGPT says: what is the data set?
MattGPT says: It's so.
MattGPT says: they have it directly
MattGPT says: records.
MattGPT says: okay.
MattGPT says: one of the types of reason why
MattGPT says: and then the areas of both.
MattGPT says: Why does the other thing
MattGPT says: try putting that you can do it? And many of you have food. Fine. Yes, we do film.
MattGPT says: And then so I can split it up. And that's cranky. But but yeah. thank you.
MattGPT says: plus dude. Bora, and then Beijing.
MattGPT says: Then you go 5 on.
MattGPT says: And then you put the functionality.
MattGPT says: Okay.
MattGPT says: by that.
MattGPT says:  no, that's good.
MattGPT says: If you add the end of that line
MattGPT says: investment fees. and I'll give you
MattGPT says: Oh, so you go to view. And so
MattGPT says: okay, so I don't know if that's
MattGPT says: in the videos.
MattGPT says: So those are the kinds of things that you do with Ecamada, even if you don't have.
MattGPT says: And then individually.
MattGPT says: and then you cross them. The line.
MattGPT says: probably about
MattGPT says: That's the level of time when the most actually, yeah. So what I would do is
MattGPT says:  you.
MattGPT says: indeed! So
MattGPT says: to the Olympics
MattGPT says: time, for example, and then
MattGPT says: then put the name and take things that don't want to score yet.
MattGPT says: Yeah, within the
MattGPT says: stop that call. Stop date time.
MattGPT says: that can work
MattGPT says: physical strength right?
MattGPT says: Thanks. Alright.
MattGPT says: take time. It's an object on tax plan.
MattGPT says: But
MattGPT says: you then at the end of that so
MattGPT says:  an adult year
MattGPT says: and you give it a pulled out just to hear from that, and then you can put it nice.
MattGPT says: and that's that.
MattGPT says: That's a way that you can add in a hexagon to this. So I would go back to the year. One thing.
MattGPT says: okay. So then, at the beginning, is it in the 90 day thing that's going on?
MattGPT says: Go back in.
MattGPT says: Yes.
MattGPT says: And what that does is that? Not here. that right? Yeah, that's a big thing.
MattGPT says: But any the idea of the booty
MattGPT says: again.
MattGPT says: I'll start to move.
MattGPT says: Yeah.
MattGPT says: see if this was
MattGPT says: go back to see that it's just too broad.
MattGPT says: Yeah. I mean, I mean, I think a topic is really.
MattGPT says: one
MattGPT says:  but it's a huge one there, right? So what it was really kind of into
MattGPT says: well, I know those that I'm trying to remember very well
MattGPT says: journal articles that I have
MattGPT says: enormous application for jobs. But right? Alright.
MattGPT says: that was just like a good little example.  So maybe if I was not just the
MattGPT says: of what it means.
MattGPT says: So
MattGPT says: yeah, I mean, you can pass that, Steve. But it's yeah. It's it's like, you know, there's there's bias in in data, there's bias in and and that, you know, you upset me all kind of
MattGPT says: the way that you're trying to see more focused on
MattGPT says: opportunities. But also then, like. what is it? And people think you love it like.
MattGPT says: it's, it's interesting, you know. Yeah, it's very much that technology cool. And couldn't we do this and this this
MattGPT says: in a way
MattGPT says: not to say that
MattGPT says: can be a woman and have the same interest and do exactly the same thing right? But at the same time it's kind of like we don't know what
MattGPT says: it's a lot. So if I can, I had
MattGPT says: had a female CEO
MattGPT says: a lot
MattGPT says: that they defined it. But actually, you know, like, what would they do? I need to ask you. That was the and the complete quality of
MattGPT says: right. So I mean, that's why
MattGPT says: that day of baronism.
MattGPT says: Look at that.
MattGPT says:  you know, they get some sort of conference.
MattGPT says: And
MattGPT says: so yeah, I get thinking about like what we really want is that
MattGPT says: is it like education? Because the last time I heard they were like
MattGPT says: at least sort of middle high school. There are more
MattGPT says: female doing stamp part of that, and mail right or thing.
MattGPT says: But then, by the time you're paying, and then, by the time again, don't market. Why is that?
MattGPT says: And so you know, you could swallow that for the thing, and then you could also like, and it's fine to frame. It has.
MattGPT says: My argument is that we just need more females in the tech industry doing this kind of work. and then it's prime.
MattGPT says: So
MattGPT says: why we have this burning. But then little daylight.
MattGPT says: what might be some of the
MattGPT says: one small holiday? Yeah. And
MattGPT says: after, I mean some of the data summit or that kind of thinking to be really useful.
MattGPT says: basic principles of like
MattGPT says: bigger power. And so, like Google Powers, great power, we can probably so much, some of them right? And then.
MattGPT says:  yeah, you know. And so they think that industry is
MattGPT says: then someone who is manually doing calculations. They tended to be like.
MattGPT says: thank you co-break
MattGPT says:  but like, you know, the pay and equality with the the opportunity. And I think part of in the end, there's a lot of things around
MattGPT says: having any to be investigated. Yeah, yeah.
MattGPT says: And this is due Sunday. Yeah.
MattGPT says: And then just for that open data set. I was trying to find a correlation between
MattGPT says: like it there
MattGPT says: metric.
MattGPT says: And it was just kind of hard to do that without suffering, separating both, because this was all senses.
MattGPT says: Yeah, and then, on average, you can't really. Yes.
MattGPT says: I changed both. But in general you can't, really.
MattGPT says: It's not that much of a
MattGPT says: yeah. So
MattGPT says: you know, first thing to get into an understand
MattGPT says: different ways. And this is a really fine investment data. And out of it.
MattGPT says: Oh, aye, goodbye.
MattGPT says: Okay.
MattGPT says: and submit from Jupiter Hub. So you're gonna use that notebook
MattGPT says: and
MattGPT says: let me know if there's any questions for that.
MattGPT says: And this quiz due tomorrow, which, same as usual.
MattGPT says: or more capital data.
MattGPT says: literacy fundamentals and learning to see data
MattGPT says: Max-heap. And it's a few questions that I've added in from that
MattGPT says: state of feminism.
MattGPT says: I think I go with that
MattGPT says: really helpful
MattGPT says: about that. Do you think about?
MattGPT says: And then the one other type of data, science and content. Just a short chapter that gives you 6
MattGPT says: different case studies, and all I'm asking you to do is to read the chapter and then pick one of them that interests you
MattGPT says: talked about applications like spelling, checking
MattGPT says: speech, recognition,
MattGPT says: data and health science in in the healthcare field and so on. So you can pick on it.
MattGPT says: So that's due Friday.
MattGPT says: I think I think everyone looks either in good shape, should be able to knock out the essay by Sunday. But let me know if there's something
MattGPT says: you put your party pick up by now. Let's
MattGPT says:  I don't, you know. I certainly don't get penalties for people who can't make deadlines who have a good reason for that. So if you are having issues, I'll just let me know. Certainly, if you like.
MattGPT says: I'm spending all my time trying to get the citations to work and they're not working.
MattGPT says: Then let me know. We'll try and figure out something from that. Okay.
MattGPT says: cool. Thanks. Everyone.
MattGPT says: Just checking in with the zoom people. If there are any questions let me know. Hey? Andy
MattGPT says: put a chat. Didn't live by there any but like just to repeat those things?
MattGPT says: Short papers you on Sunday and the quiz on Friday. Okay.
MattGPT says: thanks.
MattGPT says: Okay.
MattGPT says: Chicago.
MattGPT says: Anything going on? Hello,
MattGPT says: okay.
MattGPT says: And what we're doing and what you'll do in your your project. Okay? So, and if you haven't seen it yet, you spend some time looking at those.
MattGPT says: Those are the projects the group project years ago
MattGPT says: are pretty really good, somewhat pretty good.
MattGPT says: So don't look at it. Be like oh, that's as much as we have to do right.
MattGPT says:  Expecting you will do better than all of these projects and professional worlds.
MattGPT says: Data find it. And data story knows. Okay. But you get the idea of the kind of thing
MattGPT says: through our people, they get to a group of free or maximum 4. And that's about what kind of project using open data is.
MattGPT says: Whereas, you know, there's good open data available for a
MattGPT says: data activism for data investigation initiative
MattGPT says: project always works well. Let me know that you're going to be able to see that
MattGPT says: probably some mapping data out there. But if you have
MattGPT says: some other question or some other data set we want to work with.
MattGPT says: That's fine, too. Be careful with tidal data set, and mostly know what tag is. Right? Titled like a machine learning competition site. It's great for that. But the data tends to be pretty limited and sometimes somewhat crappy.
MattGPT says: Is there any care about that? It's all about evaluating machine learning? Right? So you may find something related to a topic you're interested there.
MattGPT says: See if you can figure out from there where it came from, and if there is a way to get
MattGPT says: more data or the more convenient data.
MattGPT says: From experience. I'm just saying people would probably put it out of data. I'm on plan.  okay, so I have
MattGPT says: how many people have decided to kind of pull the group release.
MattGPT says: Okay? So we talk. okay, cool and then you have a sense of what you might want to work on.
MattGPT says: Yeah, we're gonna do native demographics of students.
MattGPT says: Okay, so that data is available.
MattGPT says: Yes, from each point to one data center.
MattGPT says: So I like, I'm working on this data for a bit. So I gotta just put all the data in the sketches.
MattGPT says: comparing that to the others, well.
MattGPT says: okay. So then you've got those data
MattGPT says: it sounds good. I mean.
MattGPT says: yeah.
MattGPT says: that's good. Yeah. I mean, I know there's a lot of data either. And
MattGPT says: yeah.
MattGPT says: we care about looking into the same data and look at like types of prime settings on our campus. Yeah.
MattGPT says: back.
MattGPT says: Okay, so there is that those data that exist?
MattGPT says: Okay, cool. Okay.
MattGPT says:  yeah. So like, that's that kind of
MattGPT says: sounds pretty good, as I mean, you've got the time ports up there. You've got
MattGPT says: and we've got the website.
MattGPT says: is it? So it's not? And it's like a Pdf, what kind of phone? What are the Pdf, look like they're tables. Yeah.
MattGPT says: Yup, okay.
MattGPT says: type value status.
MattGPT says: And then location is an address or the auto rest.
MattGPT says: Yeah, okay? So you can use those?
MattGPT says: you could do where should you know it?
MattGPT says: But like, if you wanted to build that forward or create an interactive sort of thing that will be sort of like the the like setback device. But if you do not to this, but we're gonna formulate it in that sort of way, too.
MattGPT says: but it's not but
MattGPT says: and you could do a comparative sort of things. Tend to be with Billy already with some like you could say, like, you know, students being targeted prime, or is it general?
MattGPT says: But think about like some extensions bunch of work within the pocket but that kind of
MattGPT says: cool
MattGPT says: and you got an idea what's up?
MattGPT says: We? We are still thinking here. I think one of the ideas was quantifying
MattGPT says: the inequities between Wharton and other schools, like the number of like study rooms available for hunting money, for class sizes, for the amount of donations that go into do building construction over the years. And
MattGPT says: there's data on the study rooms available, for example, using like the pen, mobile app, or like pen libraries, printing might be a little bit more of a
MattGPT says: self created data scenario. But I think we have to consider like what data sources do exist like, yeah, that we can track the university's funding and track their donations to figure out, how do we like
MattGPT says: categorize that by the quarterback school? Yeah.
MattGPT says: okay, cool. Now, that's not kind of next. Yeah. Then you do
MattGPT says: so. Yeah, if you can do a bit, what? Just see? Like to take the source of the data. But I mean, Pdf, somewhere and reporter website, like, say, like a system that's great, for all probability right? And they save.
MattGPT says: And then there's always requesting right the background is put it on the
MattGPT says:  the should be able to.
MattGPT says: They
MattGPT says: a little bit different to a public school. I guess, in the way that they have to be available
MattGPT says: always pizza. which is interesting. And you know, I mean, for example, it's going to come with the
MattGPT says: you may find some of the data. Yeah, we get that on a day
MattGPT says: pressure. I'm getting it.
MattGPT says: You gonna work as a pop group. Oh, no, you got it.
MattGPT says: We're probably here.
MattGPT says: We're looking at like open data. Right now, I'm looking at like
MattGPT says: Philadelphia inspection.
MattGPT says: which was the product.
MattGPT says: What's going on?
MattGPT says: We'll see a family around it.
MattGPT says: Yeah. Kind of interesting to you to work on together. Then.
MattGPT says: yeah, okay, well, you can talk about if you have our ideas like, that's the idea. I want you to work. At least, I mean I did in A, in a free people who are here in a group will probably get put into say, like, I'd like to do this, and then, you know, you want to have a participant.
MattGPT says: cool, alright. So he. He knew how the booking amongst his own.
MattGPT says: Some of you, you know already seen, or have. They don't know the data? Great. That's the next step. Once you have, your idea is to say, like how easily, how you know
MattGPT says: and then as a rob guy, then, you know, yeah, kind of question like a possible idea like lens of social media. How easily available is data that's pulled from Amazon and Google, you know, if you go on Amazon or Google profiles like how they target or like advertise to you
MattGPT says: like they think you're this age group, this gender you shop like this like is that publicly available like aggregated data? I don't think so. Right? Yeah, we're kind of in the realm of like private data, private company data that
MattGPT says: you could again, you could check, you can do stuff on on the data side, right?
MattGPT says: Google data search. Amazon. Look on it. Aws data. Set the whole bunch of data set.
MattGPT says: But I'm guessing, that is gonna be, even if it's anonymized. It's gonna be a private data.
MattGPT says: Okay?
MattGPT says: That's great idea. Get everyone in the data and other people right?
MattGPT says: So in general, like social media stuff is a little bit tricky.
MattGPT says: Amazon reviews. There are data yelp reviews. There are days that have been released. That sort of thing, you know. Yeah, so.
MattGPT says: But you know, people we have done. People have done stuff with things like reading from
MattGPT says: tomato, etc., like, although we can't do that kind of thing.
MattGPT says: then you can.
MattGPT says: Yeah, something like that where the data available
MattGPT says: cool. Okay? So yeah, people can keep thinking about that. And then we'll check in on Tuesday again, about this right? And then I'll ask you to kind of
MattGPT says: this is our group. This is our idea at the moment. because the data set you can fill that right? So
MattGPT says: a good, you know, is is to have a poor data set like, say, the the habit of data as well as data. Politics has demographics in it. But then you also want to find
MattGPT says: our data center are related right? So as a rough guy, you want the main data centers. You have people in your group at the time. Right?
MattGPT says:  So if you gonna have 6 days, that's that's great. As long as you can handle it fine.
MattGPT says: Any questions about topic or about how it works.
MattGPT says: I think we hear, like Annabel and the group down the front. We're gonna work with the group as well. I'm not sure what they sort of might get.
MattGPT says: So
MattGPT says: yeah, we'll check in again on Tuesday sometimes, and then next week I'll get you to kind of send in from each group proposal and form up a group.
MattGPT says: I'll give you a shared folder, so that each group has a
MattGPT says:  not like Google Drive, unfortunately, where you know, be in the same document at the same time, it's both in the same notebook at the same time you're gonna get back, at least is away. It works okay that way.
MattGPT says:  and then the week after next, on the
MattGPT says: Thursday
MattGPT says: we'll do the budget initial presentation.
MattGPT says: Okay, so that's like a 3 to 5 min presentation for each group.
MattGPT says: This is what we're doing. These are the data that we're going to work with. Here's our plan announcing.
MattGPT says: They do that. Thank you.
MattGPT says:  Can you repeat when that is, I will. But it's going to be a 2
MattGPT says: Well.
MattGPT says: yes.
MattGPT says: 2 weeks. The day, I think is that is that across the
MattGPT says: we're in November. Then. Alright.
MattGPT says: you're second. You'll play this that November second.
MattGPT says: Yeah, okay, I'll put it on.
MattGPT says: Okay. Great. So more about Geo data. Geo. Pandora. And then,
MattGPT says: you can go. We can work on assignment.
MattGPT says: The assignment for this week, which?
MattGPT says: Okay?
MattGPT says: 50, bumps to get there.
MattGPT says: you will know.
MattGPT says: So yeah, she can look at
MattGPT says: location.
MattGPT says: That's for the 12 working with geodata.
MattGPT says: So we went over the basic concept there.
MattGPT says: Geo geopandas, is an extension pandas, Geo data frames.
MattGPT says: which is geometry. Column
MattGPT says: number. Geometry is going to be one of 3 things.
MattGPT says: There'll be points defined by a longitudinal latitude.
MattGPT says: lines which can be single between 2 points so it could be a path.
MattGPT says: and then
MattGPT says: we could take the outline of
MattGPT says: Eric, and and then you'll also become like this.
MattGPT says: So they record this point there's more dependencies. that sort of thing blind and then
MattGPT says: hold on.
MattGPT says: and then potentially have disconnected.
MattGPT says: saved by that multiplies.
MattGPT says: Those are the shapes that you get to find. Those are the shape objects that get to find.
MattGPT says: and that's really the part of well, that's the thing that makes
MattGPT says: You know, a geodata frame? Or do you have data? Paper has joined? If you come up.
MattGPT says: add those things. And now they put on 40 polygons. Well, we've got points
MattGPT says: that appoint data one. But
MattGPT says: okay, but that's how that would be. So if you want to look at the second notebook.
MattGPT says: it's Jessica that has been very important, and that
MattGPT says:  So what you're looking for is the ideal. Whenever you're kind of work with geodata, take off client. So.
MattGPT says: Darren, you had a question coming on the other day about like, what if I wanted to find
MattGPT says: from Raleigh about like district one of that? See what's up
MattGPT says: the reason that you want, and the kind of dividends you want and say, Geocon. And most of the time you'll find for their link to get how someone
MattGPT says: or go to the open data site. If you go to open data, Philly, then you'll see this whole mapping section. your new data mit data saying, I thought.
MattGPT says: okay, but ideally, you're looking at the geodesome biology  And so when we load them, then it's just geopandous, and so on, using as Gvd.
MattGPT says: and because they can read multiple different file types. It's one function between file. And then here.
MattGPT says: do that. And so we've got the Cf.
MattGPT says: and it's just rows and columns.
MattGPT says: and often, when you look at these, you find a bit
MattGPT says: at a minimum, you will have your data frame. Then we'll have a then be useful. There'll be some other categorical data in there and potentially some quantitative data in that business.
MattGPT says: The shape area length often comes out need that's just
MattGPT says: compared to of scaling.
MattGPT says: Do a ratio. If you've got them to be married or you've got count data, anyone put it confirmed
MattGPT says: and normalize it. Then you can use the shape area for that.
MattGPT says: And then
MattGPT says: that's what they look like does the blue color for the fill, and then a white line in that
MattGPT says: but just by taking the Geo data frame, saying, Don't understand. Oh, this geometry into a map of the
MattGPT says: thank you. So we have that.
MattGPT says: But we usually want to be talking about this by reading the arguments. But all of standard. What arguments that we're used to, plus some
MattGPT says: that's from that list. And so it's really just building on top of that. color is still color and the edge color gives you the outline.
MattGPT says: Okay, so so we get those?  you can. Also there is. There is a way to go through something like that.
MattGPT says: Boundaries that will say, Oh, you're not interested in printing and just in the whole shape boundaries as well.
MattGPT says: I don't tend to use that as much, because usually you're wanting to do something with the plot like Philip by one of the
MattGPT says: okay
MattGPT says: And then
MattGPT says: sort of like building that up. We've seen this before, you know. You can straight away this sort of code, count edibles.
MattGPT says: And then we've ordered, you know, then you the Plt  a function tools that you think about how to title, to add labels and stuff like that
MattGPT says: by default. It will just give you the XY. Axis.
MattGPT says: Think this is useful really, for anyone like the 1 point off the axis. You don't want to see those. You can't do that directly from the Poc. From it sub module, right? So what you have to have to do
MattGPT says: is this thing that we've seen before? Well, there's 2 ways that you can do it. But basically
MattGPT says: subplots we talk about as a way to get more of a plot on a theater thing. So we have a
MattGPT says: have a bigger, you know. and so it isn't big. and then can buy this. there'll be access there
MattGPT says: and then anybody
MattGPT says: just doing a single call, and you don't need. And then
MattGPT says: but what subpoenas allows us to do is to do multiple parts
MattGPT says: and say
MattGPT says: they, you know the not a problem.
MattGPT says: And then the number of roads, and another part right?
MattGPT says: And what that then gives you back is a couple where it's bigger.
MattGPT says: Go about typing. And then, yeah.
MattGPT says: depending on how you
MattGPT says:  Hi, I'm tabul if you if you didn't do it like that, and you've got 2 by 2. Then this will actually be a list of lists
MattGPT says: like, so you'll get like active row, active following like that
MattGPT says: there are, there are. he might answer.
MattGPT says: 0 1, 2, 1, 0, 1, one.
MattGPT says: right? 1, 2,
MattGPT says: one row and 2 things there you could say
MattGPT says: to, it's important.
MattGPT says: But you need to somehow capture these if you're gonna have there and then that allows you to take when you do your pop on the
MattGPT says:  So that's what's going on here with the subcons. Here, we're just saying, just put stuff up equals one. That's the same. I want it bigger, and it's only got one plot in it. But that's a way to get at the active argument there and then we can call that? A yeah. And you can see there's a whole bunch of functions that do things like a dot set file. You wanted title on the
MattGPT says: head off the line.
MattGPT says: hey?
MattGPT says: The other way that you can do the same thing is like here. Okay, so we've got our county spotted here. But we want to get rid of them.
MattGPT says:  you can do stuff hopping, and that has some advice, especially within multiple top. But we could just call this something like
MattGPT says:  map right?
MattGPT says: Then that will actually be the active
MattGPT says: object for that. for that we can do
MattGPT says: that.
MattGPT says: Okay.
MattGPT says:  alright. So then we've got some point data here. Often, you'll find the point. Data is part of the Csv file that you found, and it's not one.
MattGPT says: And so if we load in, this is where the prescription drug drug boxes are. You can see that we've got a hundred 15 rows. 150 drug boxes that our location and address getting safe exist.
MattGPT says:  then there is this column, which is Ugl reference column. Okay?
MattGPT says: And so this one actually
MattGPT says: at 1 point was a geodesic compile, but it's been saved as a field. Well, it wasn't geoph.
MattGPT says: It's it's been saved as a it's a regular Csv file. Okay,
MattGPT says: so this is where we have to do. We saw when you have separate columns which are numerical X and y, you can use the geo panders
MattGPT says:  points from XY function. And that's the way to get the geometry. If you find yourself with a string like this one is. But it looks like it's.
MattGPT says: you know. Data. Okay.
MattGPT says: okay, what that does is it gives us it turns this string on here. It looks like it gives. This is a string. If we look at the type of it
MattGPT says: that turns it into a shape.
MattGPT says: Okay? Or here, okay.
MattGPT says: so that's what we need to turn by proxy. Yeah.
MattGPT says: into a Geo handed data range. We're gonna hold. So I try and be consistent. And I see people do this. This idea of having a good name and then underscore. And then the type.
MattGPT says: Okay, so we've had like, so pulses on the yeah, this is a data frame contains prospective. Just use Gdf or geodata.
MattGPT says: Okay, but here's the construct the thing that construct or converts it into a Geo data frame.
MattGPT says: Geo data frame. Give it the panel data frame
MattGPT says: and then specify the geography.
MattGPT says: but it's going to make it into a co-dig.
MattGPT says: but once you run that and do plants, then the the chief would let us have a look at what's got.
MattGPT says: You can see that now we have the geometry column here they look the same. But actually this is a string column here, and then there's properties. And and so then, when you do.
MattGPT says: then it's just lazing out
MattGPT says: as big fat points like that  and so then you can do all the
MattGPT says: things that you can do.
MattGPT says: I think the color of the left
MattGPT says: don't please.
MattGPT says: Okay, any questions on that step up. So I'm fine. Okay, perfect.
MattGPT says:  we. We we talked about Crs is a little bit coordinate reference system. They vary, depending on how which particular system for building gear graph will find.
MattGPT says: so you can all find out if there is a crf that can specify. Now we'll find it. And so, Diane and this was like 4. You don't know about
MattGPT says: the Internet outlook satellite.
MattGPT says: I see. All right, there are.
MattGPT says: you just need to know what it's what. And if you're trying to bring together 2 geographical data.
MattGPT says: You want to make sure that they're in the same preferences.
MattGPT says: Hello on switches.
MattGPT says: Projection is the way in which you can
MattGPT says: go from one yeah. Rf, to another. Prf.
MattGPT says: another way to do that.
MattGPT says:  okay. And then
MattGPT says: here. simplest. When you've got regions. the polygon thing there, which is the boundaries for the M in Connecticut. And then you've got point data
MattGPT says: this
MattGPT says:  and set that up, and then you can do do a drop off the and then access
MattGPT says: box in there like that. Okay? So
MattGPT says:  that's a way that I would do that.
MattGPT says: Another variant of that that would be in some of the examples.
MattGPT says: It
MattGPT says: give a name to the base swap, or something like this.
MattGPT says: Okay? And then whatever you put. and then that's what you would specify in itself. When layer you can layer any.
MattGPT says: make sure they come up on the same.
MattGPT says: they become layers. Then you need to have that common thing next see?
MattGPT says: Oh. that bit out coming off.
MattGPT says: Yeah. So we're we're saying, Create this, create the public.
MattGPT says: And so then to get rid of the acting. You mute bases.
MattGPT says:  okay.
MattGPT says: that I have any questions. Okay, alright. So that's like, at a basic level, you know, understanding that finding data. you can find it. Geok. Finance. Go out the data in that you want
MattGPT says: worry about creating been over things and participating. But but if you have
MattGPT says: Csv file that has some geodata in it, and it's usually one way. You'll find point data addresses or something like that.
MattGPT says: Then you just go through that process of
MattGPT says: burning work from pandas to Geo pandas by specifying the geometry.
MattGPT says: in that.
MattGPT says: If there's data that you have that has addresses on it. Then
MattGPT says: you
MattGPT says: you can do a lookup from an address to get like latitude. From that
MattGPT says: you can also do the reverse process, too. Sometimes, if you want to get street addresses from points. But miracle records.
MattGPT says: But like the your budget where you're talking about it. and
MattGPT says: got like the location corner of 36
MattGPT says: fauna, or something like that. And just put that into. Do you have further? And I think you'd use
MattGPT says: and equipment.
MattGPT says: isn't that
MattGPT says: okay? So then,
MattGPT says: this gives us you know, visually wouldn't do a lot with that right. But often, you know, you've got data. You've got point data which is really, really dead.
MattGPT says: and so, yeah, there are 2 options, the hanging one I mentioned before, specify on your spot data transparency.
MattGPT says: which is like, the output goes between 0 and one client, one being okay. we did something like that. Well, actually, I mean.
MattGPT says: okay, you can see
MattGPT says: you can see that the point.
MattGPT says: Then just the fact that exploding them.
MattGPT says: they find out more. So if you've got really 10 point data, play around with doing it
MattGPT says: by playing out of apple visually to do that. But really, what we want is that we want to use whichever unit in our base map.
MattGPT says: Yeah. Boundaries of pound
MattGPT says: could be different, or any other thing
MattGPT says: we want. We want some way to go through and say.
MattGPT says: how many drop boxes here? How many drop boxes here, how many drop boxes here, and so like, do a tap. and so that's when you've got 2 data frames like this. That's where Geo. Hand has had a facial what's called a.
MattGPT says: and so that's like a merge that we've seen before, sort of saying sympathes emerge
MattGPT says:  in the sense that you have a left data frame and a right data frame. But you don't need to apply phone call because it has to be to Geo. Plan of their geometry. Right?
MattGPT says: I'm looking to be there. Any one of these briefing
MattGPT says: and Wednesday people join on that? Then you specify how specified, done, sequel or anything like that with joints. And you know that story and a right joint in our joints, all of those things.
MattGPT says: Basically, that means, let's you know you're starting with one less data frame.
MattGPT says: and let's say that has 50 rows. and you're merging it with
MattGPT says: one over here.
MattGPT says: 27 right in. In a regular joint. You've got a common column
MattGPT says: in a regular that join that will just keep all of the rows in here.
MattGPT says: Any other India, but not. India will get this guy
MattGPT says: right?
MattGPT says: Right.
MattGPT says:  or you can be in second
MattGPT says: failing.
MattGPT says: So
MattGPT says: that's what we're doing. We're saying here. We want count every single town right? So that's how that. So we're doing a left joint. And then what's the function here? So there's different functions that you kind of like you have
MattGPT says: geometric functions.
MattGPT says: and the simplest is just is the point, even the point contained within the problem.
MattGPT says: So that's what contains 8.
MattGPT says:  so if we run in and take a look at what we get here.
MattGPT says: you can see that we've got 190 rows.
MattGPT says:  50
MattGPT says: repeat.
MattGPT says: and then.
MattGPT says:  so the town, the town number, and then right at the
MattGPT says: and 5, are they ranked this past
MattGPT says: account data.
MattGPT says: Yeah. Okay.
MattGPT says: there we go. Okay. So you see, then, what we end up with
MattGPT says: is most of these detail. We have
MattGPT says: and boundaries over here.
MattGPT says: And then we had a list of the common unit
MattGPT says: technician. And then that right button told you how many doctors down within?
MattGPT says: Thank the
MattGPT says: we've now got, and the geometry is still the shape. But we now have a numeric column count on added.
MattGPT says: So then, what we can do
MattGPT says: something like that
MattGPT says: where we we default, and instead of specifying color, the fill with them spread by the column, and the column is a facility column.
MattGPT says: And so it's numeric, right. But after we look at the range there's there's just like 4 values, there being a no longer positive one
MattGPT says: to a prison officers. So that's what the categorical
MattGPT says: column does here. Right? So that's a full.
MattGPT says: So if we've got count data
MattGPT says:  oh, yeah, we don't have a record.
MattGPT says: Okay? So if you give it a column which is new merit by the phone, it will treat it as a continuous value.
MattGPT says: And you'll get a
MattGPT says: yeah. But in this case there's really a continuous study, just like how many drop boxes and putting it into bin.
MattGPT says: So
MattGPT says: that's why I would set categorical food for this kind of data.
MattGPT says:  a little bit more going on in that box thing. Now that we see key thing to do, the plot and emergency frame specify the column you want to use to fill, how you want it to be treated either as categorical or continuous.
MattGPT says: and then you are legend. And then this dictionary that you see here under the legend, keywords.
MattGPT says: Okay? So I guess if you look up legends and map of it. And you see, there'll put arguments that you can give legend function about like where to occur on the what
MattGPT says: and bebox the anchor thing. It's useful, but it's often led to hold them to get put within the active line. and so that'll kind of problem stops.
MattGPT says: So you can control how far out by the end region, how high?
MattGPT says: Okay?
MattGPT says: any questions on that.
MattGPT says: Okay, so hopefully, like that gives you an idea of how all of that stuff works.
MattGPT says:  let's go to one scroll down, even though it's 5, and I will. That's not really relevant. I don't think for that box.
MattGPT says: So here's another geodesome right? This is.
MattGPT says: this is us states. and this is very common, and you keep finding safe math around great, but, like Alaska, gets spotted like
MattGPT says: those in place.
MattGPT says: and it makes it difficult.
MattGPT says: Really common thing to do is to sort of break geography and move Alaska down here.
MattGPT says: Thanks,
MattGPT says: And so
MattGPT says: You'll see that there's 2 versions in here right there's this one which is us states. So do Jason. And then one. Event
MattGPT says: broken in a sense by moving going here around. But it's made them much easier to do
MattGPT says: blank
MattGPT says: if you're working with state data. These are 2 geodes that we can use for that.
MattGPT says: But then
MattGPT says: survey, which is level data.
MattGPT says: Okay, so what you want to do is, think about. Okay, I want to block some of my
MattGPT says: categorical and quantitative data onto this map, perhaps.
MattGPT says: And so let's load in and have a data set, which is us broadband access. And okay. So you can see this like state names. Got the census area
MattGPT says: 2, 2 letter abbreviation population
MattGPT says: and so on. So this is all from the sense of the Acs data is really, really ripped. There's a lot of fucking. Yeah.
MattGPT says: But one of the problems that I have is bull band academic.
MattGPT says: estimated by percentage.
MattGPT says: Okay, And so
MattGPT says: when you have those 2 things right, what we need to do is
MattGPT says: is, join the join the 2 together, right? So join or merge where we've got common problems, and these have to have a problem to join. We'll put it together
MattGPT says: so that we get both the geometry and
MattGPT says: quantitative data in there as well.
MattGPT says: Okay. so a thank you to a thought
MattGPT says: by then. And again, indeed. full percent is a percentage of
MattGPT says: okay, and so that problem gets in your problem out
MattGPT says: right here.
MattGPT says: And so
MattGPT says: things that do to make it kind of prettier.
MattGPT says: first will be, let's get rid of the active clients. Okay, so that's what for that map of the main.
MattGPT says: and we're up
MattGPT says: and you can specify the bigger size up here as well. But let's make this a little bit. Yes.
MattGPT says: okay, probably full type, right?
MattGPT says: and
MattGPT says: So you wanna you might wanna control, though, is the it's the range here. Okay? So this will give you like maximum value on your variable, plus a little bit.
MattGPT says: If you wanted to control that, you can use B min and bmat
MattGPT says: they beam in. Let's say that 60 and be maximum 100.
MattGPT says: And so then we change the scale on that, and I change how it comes out.
MattGPT says: Remember the column app.
MattGPT says: You can change using cmaps
MattGPT says: that means adopt means low. Let me high.
MattGPT says: I remember, like the little
MattGPT says: have to get at like, what color maps can I use. Just
MattGPT says: jump into the cmat argument
MattGPT says: for an error. And at the bottom of issues, Elizabeth
MattGPT says: of you can Google these or look on in the documentation about what they're
MattGPT says: play around with them. But basically, you know, once you figure out what it is which one you like, people always got the option of reverting it
MattGPT says: one direct reality.
MattGPT says: Those are our parameters.
MattGPT says: And then the last thing I want to do is
MattGPT says: this
MattGPT says: the last argument people really use by the journey is
MattGPT says: that can. Kwd.
MattGPT says: that's what people are.
MattGPT says: Okay. Now, what legend keywords is is a dictionary, right? Remember our dictionary key value. Pad but find additional arguments and things that we want to add more about electric
MattGPT says: payment. So
MattGPT says: you can look up in the documentation what those are. But things are really useful. One is shrinking.
MattGPT says: and that just says. Hi, I'm Gavin Hector.
MattGPT says: So point 5. We want the 11 features.
MattGPT says: and then you can pull it tight that way. You get something to look at
MattGPT says: time for coming out.
MattGPT says: and you can add a label.
MattGPT says: Aye.
MattGPT says: enables you to make
MattGPT says: right
MattGPT says: households.
MattGPT says: Thanks.
MattGPT says: meters.
MattGPT says:  If for some reason you decide. you don't do it that often. But if you wanted to have it go
MattGPT says: orientation, I want to get out of hand.
MattGPT says: So you got the ideas like a lot of flexibility to create these kind of problems.
MattGPT says: Okay, any any questions on
MattGPT says:  I'm sorry.
MattGPT says:  But there's one other. There's one other function that I'm think it's useful. Let's say we've got Geo data state right? Like this
MattGPT says: and like in the in assignment 4, where you had regents like in a mid-out
MattGPT says: as a another column, and some of the data analysis would be doing you doing group by and then selecting right?
MattGPT says: The equivalent in geopanders of doing that.
MattGPT says: Is this all? Okay? The function support. And then you specify it by argument.
MattGPT says: And what does that do? Well, that didn't take the polygon here.
MattGPT says: but each state and someday, which is the boundaries of each State.
MattGPT says: and then it tries to make those into a single. Next thing it'll take. Say, you want to have a map over the region
MattGPT says: instead of.
MattGPT says: And so you see outlined. As long as you add into your geodatrame a column that specified this state belongs to this place and this state and this state, A, BCD. Would be anything like that.
MattGPT says: Thank you. You know. Back.
MattGPT says: U.S.A. es salt is Old
MattGPT says: 5, whatever that come in and you'll get a smaller data frame, which flat down.
MattGPT says: but it's it's taken. And
MattGPT says: so that's that's another one that could be a little
MattGPT says: okay. Cool.
MattGPT says: hello!
MattGPT says: So that that old stuff on to your hand is, I think we need to know. I think it's like superfund to work with, even
MattGPT says: give it back. Aye.
MattGPT says: then.
MattGPT says: so yes, just for a second you can talk amongst yourself about your projects if you want. I'm just going to try, and I'm just releasing you, and then we'll spend the rest of time. Just looking back. Okay.
MattGPT says:  50 value.
MattGPT says: Oops.
MattGPT says: you guys
MattGPT says: well, let's go.
MattGPT says: well.
MattGPT says: and now
MattGPT says:  or in the presence of the day.
MattGPT says: Hello!
MattGPT says: I tell you.
MattGPT says: I don't know are exactly
MattGPT says: that was the right one. So oh, did you get a pick one?
MattGPT says: Yeah.
MattGPT says: yeah, I think it's gonna keep all about
MattGPT says: which is painted videos.
MattGPT says: The how about this?
MattGPT says: I think these households PURB. URIS. E. So I'll meet you after
MattGPT says: that's like that's like the
MattGPT says: then again.
MattGPT says: okay, cool. Sorry about that.
MattGPT says: And
MattGPT says: looking at the back.
MattGPT says: I think I put 2 on Sunday. What is today instead of yesterday? Any times?
MattGPT says: Thankful Matt, as we've been doing right. But, Simon.
MattGPT says: we have a whole bunch of questions in, and then you've got empty numbers to a blog code where you're going to do a blog post based on something out of using these data. So the data we've got here as well, bank
MattGPT says: data.
MattGPT says: global data that has Gdp life expectancy and some other practice in it
MattGPT says: at the country level.
MattGPT says: Okay?
MattGPT says: so. But and also questions focus on just doing some analysis of the the data frame understanding, inspecting the data.
MattGPT says: you know, figuring out the by region things like population, life, expectancy. and from mortality those sorts of things creating some bugs.
MattGPT says: Okay? And then
MattGPT says: questions. 1314, 15,
MattGPT says: use geopampers.
MattGPT says: And so there, there is a  a world map
MattGPT says: with comfy. Bad reason.
MattGPT says: The map within the world bank data.
MattGPT says: Do a popular look like this? Okay,
MattGPT says: And then
MattGPT says: do that
MattGPT says: collapse. Collapse the country by region. Okay? Because in the Neo data frame that you'll get a whole bunch of different components of country name kind of proposed by
MattGPT says: and then for regions, one of those partners will region underscore. Wb for World Bank, so that massive have a few days effect.
MattGPT says: Okay, so this person is just taking gait range
MattGPT says: half month. So it comes to we'll start back again so that we go to
MattGPT says: region level, and then then you'll have 2 geodespring within one that has regions.
MattGPT says: And then for question 15, is this going to do some stuff with life expectancy?
MattGPT says: And this is time. Thursday. It starts with 9, 9, 9, 15,
MattGPT says: and create
MattGPT says: block. That shows
MattGPT says: budget parents in the country.
MattGPT says: and you want to put them on the same scale. So you're going to create the app, you
MattGPT says: Kalamat. I have the range of the Kalamat. and then pop them there
MattGPT says: and then
MattGPT says: use the region that one distribution
MattGPT says: in question 10, you'll calculate percent mean percentage change by region in life expectancy. So the whole one country, they have life expectancy 90 and 5 10.15. So you want to have to extend change
MattGPT says: the average.
MattGPT says: Give you a chance to practice some of the stuff that we just talked about. Those are the questions right? I think most people, they just a lot of repetition. By now you could be feeling really comfortable with
MattGPT says: kind of doing these there
MattGPT says: using group by using value accounts.
MattGPT says:  there's often question 15. And there's some code just examples that run all those else these could be useful using different ways to do time theory possible.
MattGPT says: It's it's good thing. And then vegetable food.
MattGPT says: Yeah. So Google is what I'm looking for. You've got some comments on 4, is it? Should be stand as an independent
MattGPT says: you know, as an independent
MattGPT says: that you can bring the data in. Have a post up there, but don't just have a whole bunch of code sales run and then have, you know.
MattGPT says: few Markdown sales, and maybe one clock something about that underneath. They want to see integrated search on the
MattGPT says: parallel to braming things. What are the interesting questions? Right? You can model that on a good piece of data, journalism or data journaling. Marco?
MattGPT says: You know, you might.
MattGPT says: But then bring the reader in. This is a really interesting question that you don't have to cover, and you shouldn't try and cover everything that's covered in the assignment. So pick something
MattGPT says: that's very interesting. There, combine that
MattGPT says: you want. You know, you want to be creative, really?
MattGPT says: You know, country data.
MattGPT says: something about these countries which isn't in the data set there managed to bring that
MattGPT says: and then you've got the total that you need to create a different type of block.
MattGPT says: You put in a block. It doesn't speak the stuff. Okay? So in the block, you wanna make sure that we set it up. We have blocks and have at least a sentence or 2, which is like, what do I see that? What is the observation for the people?
MattGPT says: Okay, so the biggest change and life expectancy.
MattGPT says: Where has it been? Qualified right? And then
MattGPT says: you wanted to have a poll about that.
MattGPT says: you know.
MattGPT says: Okay, so
MattGPT says: yeah. go ahead. We've got those frame assets. But you want to start working on that and have questions.
MattGPT says: You wanna talk to me about your project ideas. And having to do that, too.
MattGPT says: Have other questions.
MattGPT says:  I have a question.
MattGPT says: Okay?
MattGPT says: So what? There's one line for every from 2022, 23 year all the way back to yeah. Yeah. So I just selected.
MattGPT says: did data about like.
MattGPT says: like, if I want to make like,
MattGPT says: yeah. So
MattGPT says: we're gonna like, somehow invent memory from. Oh, it's been cool.
MattGPT says: is higher.
MattGPT says: But then I think she's really
MattGPT says: yeah. So you can't really have any information about like.
MattGPT says: do we have like like the
MattGPT says: like? Should we just aggregate all those to get get the like student of color over.
MattGPT says: So that's okay. Right?
MattGPT says: Alright.
MattGPT says: the
MattGPT says: yeah. But I mean, the main sort of question is what I'm playing.
MattGPT says: So we need something
MattGPT says: snapshot.
MattGPT says: Yeah, I feel about, you know, zoom right now we have 3 schools with 2 data sets.
MattGPT says: Since 6 like. And then I just did a average.
MattGPT says: Yeah, so like. see that? Yeah, I mean, and you should. You can kiddo that kind of
MattGPT says: are you happy with it.
MattGPT says: Yeah.
MattGPT says: what else can we do
MattGPT says: beyond just things that? So
MattGPT says: thank you. A lot.
MattGPT says: Think about like, what other data I said is what I mean?
MattGPT says: It's all like public information.
MattGPT says:  okay.
MattGPT says: that's right.
MattGPT says: Yeah. It's like.
MattGPT says: alright.
MattGPT says: yeah.
MattGPT says: Depends on data.
MattGPT says: No.
MattGPT says: What? What would be lots of dying?
MattGPT says: Hello.
MattGPT says: yeah. what? We and circle for 8 h.
MattGPT says: That's what we know.
MattGPT says: Okay.
MattGPT says: it's always fun. Yeah. Good idea to help.
MattGPT says: I presume that every day of Thursday by
MattGPT says: yeah.
MattGPT says: Oh, how do you find that
MattGPT says: I get 7 data sets for coupons?
MattGPT says: So what would be the kind of questions that you'll be asking right? Because you find it like, kind of dying.
MattGPT says: Oh, yeah.
MattGPT says: and figure out what the number find it. Online menu.
MattGPT says: and then what is?
MattGPT says: Oh, actually
MattGPT says: year 7.
MattGPT says: And then if you were able to buy them for a Yale average?
MattGPT says: Yeah, that's already well, okay.
MattGPT says: But during the sun.
MattGPT says: so
MattGPT says: not really.
MattGPT says: There's been a trend.
MattGPT says: But now that will do the time. Gonna have to not do this one right now.
MattGPT says: I said. They were like, you know, we don't have to wait till looks like it's going to at some point.
MattGPT says: Yeah.
MattGPT says: But I'm able to to get that into. I'm just working on the yep.
MattGPT says: If you go through, they only release
MattGPT says: not ideal. So we're like experimenting around with ideas. We're looking at like universities. 9 90 forms. Yeah, there's anything interesting there that I kind of looked at. But I mean as well.
MattGPT says: 15 bucks. Oh, is it? Yeah.
MattGPT says: like records with us? But yeah.
MattGPT says: that's
MattGPT says: that's some transparency. Yeah, exactly. Yeah. But if you need that way, the data comes from, yeah, cool. Alright. So I did try to get the data I contacted like the Admin as well, website.
MattGPT says: and actually, the data access grants are actually currently on hold.
MattGPT says: Oh, okay, yeah, do you know, if they have a policy against scraping. I don't believe so. I have no clue. I asked if I also asked if I could just like contact one of the
MattGPT says: a former like person that reached out for data just like recycled like whatever they they were given that
MattGPT says: and they're like that will be frowned upon because, like each like data, access request is like custom done, or something like removing ids from people's names and stuff something weird. But
MattGPT says: I think I will be able to scrape. but it would just be like
MattGPT says: if I did script. That would have to be like entirely from like my point of view, though right like it's like you have to make an account to play. So all the all the competitive matches would just be from my account. So there is a white C.
MattGPT says: But then, unless you you kind of see, like a history of all games that have been played. you can. What is the sign of like, yeah, let me
MattGPT says: output. Audience. oh, sorry. Yeah, there's a second. I have 2 other ideas, but like, I think one I'm going to save for my our class.  we've got signed in.
MattGPT says: or family or not.
MattGPT says: This is normally so. Watch that all. Okay. So if I go to watch a battle. It keeps like
MattGPT says: 100 plus right here, my computer will also. So and so that that text export that you send me that you get that you download that from here. That would be yeah. Or actually.
MattGPT says: yeah, I'm sure you can do it slow right now.
MattGPT says: Wow. oh.
MattGPT says: okay, maybe it would be better.
MattGPT says: Yeah.
MattGPT says: sorry. But you can just come here.
MattGPT says: Okay, so what is it? Real? Replay would not
MattGPT says: sure. Actually, they say, so please go. Come on
MattGPT says: right?
MattGPT says: Hang on.
MattGPT says: Yeah, all overcomp.
MattGPT says: I believe that that should be it.
MattGPT says: Nope, that looks like it's not that. Yeah. You go ahead.  okay.
MattGPT says: take a second full month. Yeah, it should just be this replays.
MattGPT says: That's interesting. Did II don't know.
MattGPT says: It was like, I actually feature replace
MattGPT says: featured as stuff performance. That's not scalable.
MattGPT says: I have no idea how it's expensive. Older stuff.
MattGPT says:  okay, so then let's say so. You were getting like you click on you. Click on one of these right?
MattGPT says: Yeah. And then it just plays a match.
MattGPT says: When sorry. Let me turn that using the call
MattGPT says: alright. There we go. Actually, it's got a big sound effects, too. By the way.
MattGPT says: there we go.
MattGPT says: They're looking for number 11, number 11, asking questions of data.
MattGPT says: See that?
MattGPT says: Okay? So this kind of but reminder where we are to. You know
MattGPT says: those 2 things together. Make up that great for that.
MattGPT says: You're gonna have assignment 5,
MattGPT says: not this week, but next week. And that's what I was just describing. It's just going to be this one which is a little less structured, I mean, I'll give you structure, but it's not the same sort of questions.
MattGPT says: the same questions that everyone's working on with test cells gonna be going through the initial data analysis steps with a data sheet with a data set. Okay? And doing that.
MattGPT says: giving answers to that. So that'll be assignment 5
MattGPT says: that will take through next week, and that'll be due on next Sunday.
MattGPT says: Next Tuesday is Halloween.
MattGPT says: and so I love how we
MattGPT says: please.
MattGPT says: and I don't know.
MattGPT says: People dress up to go to class. But why not?
MattGPT says: I will do my best to come up with some sort of costume, which is data beam. Okay. it'll be worth coming to us just for that
MattGPT says: But then
MattGPT says: At the end of next week we'll start looking at heart. Okay, so
MattGPT says:  everything. That kind of we've done so far and pipe. And then, you know, all of that will transfer over especially what we've been learning in pandas in terms of
MattGPT says: ways to work with popular data. We just spent a couple of weeks looking at our
MattGPT says: so that you can see. Yeah, you know, some people use are, and some places that you may end up working or projects you may end up working on.
MattGPT says: They may already be using our okay. So it's good to be a little bit bilingual when it comes to data analysis languages, just python, and are on the main button, because I want to give you some
MattGPT says: exposed to that in the most popular format for doing that which is a
MattGPT says: or sort of library than article tiny Births, which is a way of
MattGPT says: working with data enabled in a very sort of systematic pipeline way, and a plotting library which is called gg plot. And so, if you should notebooks to do that will have assignment, 6 will be based on
MattGPT says: using. To do data analysis.
MattGPT says: plotting, using those.
MattGPT says: And then after that. So that takes for assignment 6. And then you've got your quantified self assignment, and that will be using those tools from R to do the quantified self.
MattGPT says: Okay, that then sort of takes us to the end of
MattGPT says: that would be the end of the assignments and the second main assignment. And then the last thing left for the class, then, is the data project. Okay? So after that we'll talk about that. And that's what you focus on
MattGPT says: the semester. Yeah. So  let's take a look at, sir.
MattGPT says: Let's take a look over 11. Okay, just asking questions of data.
MattGPT says: So there's but there's 4 notebooks in here. The first one just summarizes what you've already read. from
MattGPT says: in Chapter 8 of data, literacy fundamentals.
MattGPT says: Okay?
MattGPT says:  But the way, where? Who? When, which? What? How? How much time?
MattGPT says: Okay? These were. Remember, this was chapter 8, and Jones is using that number of the number of his chapters, the number of points kind of thing. So there's probably too many questions here.
MattGPT says: just because he was trying to get 8 out of there, but certainly
MattGPT says: the first 6
MattGPT says: key ones, right? And all the W ones. Okay. So
MattGPT says: when we come across a data set that we're actually looking at. So
MattGPT says: why does this matter? Why am I looking at this interesting like a number of wraps in New York City, or
MattGPT says: the amount of pizza, the the distribution of pizza use over pizza eating London over the semester by your average undergraduate favorite candy in different states in the Us. And with that candy data set that we looked at. Exos can sometimes be a little bit on this kind of in that.
MattGPT says: But you want to think about like, why does the data matter to you?
MattGPT says:  and usually that will be put in some sort of task that you've got, or some sort of question that you want to, that you want to answer. And so it's an important issue. Something that you're curious about.
MattGPT says: Think about that right and be, you know, it's
MattGPT says: it's a
MattGPT says: important to to understand why it is that the data that you're looking at is important.
MattGPT says: For you have property.
MattGPT says: And especially, remember, data are people or people become data or parts of people.
MattGPT says: things that people do become, even because we always want to remember that things that we have there.
MattGPT says: Then where did the data come from? And so we put we talked about this number of times. So the provenance of data and sort of the issues that, and in the quiz last week, when we were looking at those 7 sort of principles of data feminism.
MattGPT says: you know, I should pick one and think about a particular data set. And I think that's a nice sort of so the principle of the questions that when it, when you're thinking about something like the shootings, incident or crime databases that we've looked at.
MattGPT says: those are kept up by the Philadelphia police department. Again, it makes sense because they respond to crimes. They record them into a database.
MattGPT says: Okay, but
MattGPT says: that's where they come from. Okay? And there could be another sort of shooting.
MattGPT says: shooting incident data that comes from an independent community-based organisation or source
MattGPT says: and understanding the difference between those and what, how that may impact, how might that impact, how data is tested, and how it's managed.
MattGPT says: how it's organized. and that then fives into the
MattGPT says: the who part of it.
MattGPT says: hey?
MattGPT says: who phones the data and who updates the data? Certainly in the private data realm that we talked about. You know all of our data that we we may think. And this is Tension window with the
MattGPT says: personal and the private.
MattGPT says: We use Gmail. Most of us probably use Gmail, or something like it.
MattGPT says: We use social media, we write things, we think those are our messages. Those are our data.
MattGPT says: hey? To to the degree that they are. But really, who? Who is the day? We don't own that date.
MattGPT says: If you use Gmail, you do not own the data that you produce the emails that attend. If you use Facebook or Instagram or desktop, you don't own any of that data
MattGPT says: terms of service which we just kind of said, yes, yes, when we install the app to
MattGPT says: there who owns the data?
MattGPT says: Is there?
MattGPT says: the company or the organization that actually sort of provides that service to us.
MattGPT says:  so we always want to think about that when we're looking at a data set. Where did where did it come from?
MattGPT says: Okay,
MattGPT says: And then who owns and updates the data? And a lot of that data that we've seen on like open data. Philly, we've noticed like there's missing pieces. Or it's great. There's really interesting data. But it stops at 2019,
MattGPT says:  or something like that. Okay? So then, when
MattGPT says: when was the data love data? How current is it? And then this is part of what we've already been doing when we do the exploratory data analysis.
MattGPT says: what does each row represent? What are the columns we pretty much can see, you know. Sometimes you see a lot more columns in there that are not that valuable to to you or not that interested for your questions.
MattGPT says: But come with the data. Okay? So figuring out which of the most important variables is going to be tasks or questions to go in there?
MattGPT says:  and then what are the definitions? It's really easy to just kind of skip over this path when we look at the data with their effect.
MattGPT says: But yeah, bring ethnicity age gender. Alright. Okay. Those demographic variables make groupers together and then move on. Actually look a little bit more carefully today when there's a comment, is labeled Agenda, hey?
MattGPT says:  what? Is it? Just a binary option there? And actually, is it sort of asking or defining sort of like a verbatim sets there, or is it gender identification? What are the options that are in there? And I think we looked kind of a while back at sort of
MattGPT says: or in the
MattGPT says: it's in the readings talked about.
MattGPT says:  Facebook kind of updating is
MattGPT says: self-identified sex or gender variable
MattGPT says: to allow people to enter multiple
MattGPT says: self-reliance. So I didn't like that. What was it?
MattGPT says: Okay? So it's important when we're thinking about?
MattGPT says: when we're looking at the data thing. What are the definitions? Okay, not just
MattGPT says: what is the label, but actually
MattGPT says: is there. And that's where we'd go to the Metadata dictionary. And look at that. And
MattGPT says: okay. And then, so how is the data measured? Stored?  again, have a very data set data setbook
MattGPT says: things like shootings and shooting Internet database. We look out. Well, data is measured, presumably at the response to the incident.
MattGPT says: Police officers go out to the shooting incident. They've got some sort of maybe paper. But these days on an ipad or similar table tablet, like thing.
MattGPT says: where they record all those data. So that's how those
MattGPT says: variables that measure. And they have forms that fill in for that.
MattGPT says: Okay, so
MattGPT says: how many data that we get maybe more objectively measured. It makes sense the data.
MattGPT says: It may be log data from interactions on a website. Okay, that's just every maximum visits a website, you get a big timestamp and which Paley just visited.
MattGPT says: But then, on the page. That sort of thing. Okay? So that's measured for
MattGPT says: automatically
MattGPT says: and then
MattGPT says: related to that. Then
MattGPT says: how are those factors we have most of the time. There's probably some form like in the case of the shooting incident. There's a form that then goes into
MattGPT says: database
MattGPT says: and stored in those. By the way.
MattGPT says: Thank you. Macy's.
MattGPT says: When we looked at the police shootings
MattGPT says: data, we looked at one of the key variables which webinar officer was involved in shooting in the shooting or not. We noticed that there were some differences on the demographics, so that the race and ethnicity of the victim was not recorded.
MattGPT says: When an officer would vote. That may well be there
MattGPT says: may be measured may well be collected.
MattGPT says: but it doesn't. It's not stored, at least in the format that we were able to download. So those are interesting, important questions.
MattGPT says: and then there's last one that he includes there, and a little bit obvious. How much time do you have to consider the data? Sometimes we just have a quick question about something. You know, we're writing a paper for another class, or we're in a work context. And it's like, Oh.
MattGPT says: it would be really interesting to look at the data for that right if you're writing, and if you're waiting for the Vp. Or you write an article, you could just quickly go off and do some quick data analysis. But you don't have much time to look at it. You just want to get a quick answer to that. But other times, like for your data project in this class, you've got weeks to look at this data and to really consider it. So that allows you to expand the questions that you're asking.
MattGPT says: Okay? So these are important and reasonable
MattGPT says: questions to ask, and this will be part of what I'll ask you to think about for the data that you pick for
MattGPT says: the assignment. Okay,
MattGPT says: then this this one. There's some extra bits of pandas that
MattGPT says: we haven't.
MattGPT says: that we have kind of covered, probably, and that you've seen in some of the the workbooks. But I wanted to sort of go over
MattGPT says:  So if you just want to open up the second notebook and run through it.
MattGPT says: the setup here. Okay, we're gonna load this datasheet.
MattGPT says: which is the measles cases.
MattGPT says: this is world health organization data. Okay, so meet those cases by country by month.
MattGPT says:  okay, so these are just the steps that we're doing, load the data in
MattGPT says: and then looked at
MattGPT says: inspector data here to get a sense of what it is of what it needs road.
MattGPT says: Each road seems to represent a country
MattGPT says: a year, and then those are the 1, 2, 3
MattGPT says: we got 3 identification variables, country country code. And then we've got a region world. Okay? So these 2 are the same, but in a different manner.
MattGPT says: Weekend. And then we've got a time. Locator.
MattGPT says: Here, then.
MattGPT says: and then we've got
MattGPT says: you have data number of measles, cases recorded. Sometimes they would have to look the metadata where these are actual.
MattGPT says:  cases, are these patients millions or thousands or something like that? So we want to take that. But let's just pick them as cases, I guess an ounce, and then you can see it per month by month in that year.
MattGPT says: So pretty
MattGPT says:  cleanly structured data. Just by looking at the top, you can see it's going alphabetically. But like, I don't see any missing data yet in that So we just can then look at the columns like just to confirm what we've got there.
MattGPT says: Think about a few weeks. What kind of table is this? What kind of data is this? Is it wide, or is this long
MattGPT says: kind of fall? And how many people think long
MattGPT says: back, and
MattGPT says: how many people mean white?
MattGPT says: I mean, I can see why you think that like this is
MattGPT says: this is, remember the distinction between wide and long and why data. You have
MattGPT says: the same.
MattGPT says: You have repeated measurements in different units here. So the unit here is in a month, and the measurement here is the number of cases of pieces.
MattGPT says: But if you have that repeated across
MattGPT says: multiple columns, the same measurement, but just by a different unit. Yeah, which is was not. Then that is why they go.
MattGPT says: And you can see reading this if we're looking at patterns. If we just wanna go. What's going on in Angola in 2011. This is a really good format for so usually white data like this. We see when it's actually presenting tables that we're gonna look at.
MattGPT says: because we can then scan from left to right across the year and see what's going on?
MattGPT says: Okay? So then, the long data is much, much longer but narrow. So what that would do would keep these free.
MattGPT says: These 4 columns the same. Okay.
MattGPT says: we'd have these forecasts to feed. Identify the country in the year, which is what each row is.
MattGPT says: And then we have
MattGPT says: one row for January, and then the same. 1, 2, 3, 4, and then February. So we go from having one row per country per year and having 12 rows per week.
MattGPT says: Okay. So each row would then represent a country in a year and a month.
MattGPT says: That's long data there.
MattGPT says: Okay, so sometimes we want to work with long data. Sometimes we want to work with wide data
MattGPT says: like, say, looking at this visually. this is good
MattGPT says: for getting a mean
MattGPT says: in a year. This is, it's possible to do that with this, but it's a little tricky, because we have to say January plus February, plus March, April, etc. Etc. Divided by 12.
MattGPT says: So we have to sum across a whole series of columns, and then divide by the number of
MattGPT says: which we can do in pandas. But it's easier to have it in long form, because it's easier to group by rows and then to calculate a mean.
MattGPT says: okay.
MattGPT says: so
MattGPT says:  just to sort of clarify that they
MattGPT says: we want to go from. We want to try and see how we go from wide data to long data. Thanks.
MattGPT says: So we want a list of the columns which we have, which all the actual data is the number of cases of meters. So those are columns 5
MattGPT says: through the end. Okay, so there's Polycom. We can just slice those out
MattGPT says: and then we have work for the identifying columns or the Id columns. Okay? And that's the thing that stays same when we transfer Y alone. The region, the Iso code in the country, those 3 already
MattGPT says: the one way of identifying the country, and then the year of June, the first.
MattGPT says: Okay?
MattGPT says: And so there's a function called melt.
MattGPT says: And so what Mel does is it takes wide data. turned it into long data.
MattGPT says: Okay, so
MattGPT says: this is an example of what this looks like. Right? Yeah.
MattGPT says: which we. Gonna
MattGPT says: and then without a swing from having 12 separate columns for the master having
MattGPT says: one column or 2 column, one identifies a month and the other that identifies
MattGPT says: Patrick. Thank you
MattGPT says: that. And we've got our id column.
MattGPT says: Yeah. And then
MattGPT says: January, December, hey? So this that's the Y version.
MattGPT says: What we're going to go towards
MattGPT says: when we mount it is that we keep the Id columns hand.
MattGPT says: But then we have just 2 columns.
MattGPT says: So we have the for the variable and the value. Okay, so the variable comes from the column name. So we'll have
MattGPT says: just goes over again to January in a row.
MattGPT says: and every
MattGPT says: December. And then the value here that goes to that.
MattGPT says: Here's for that.
MattGPT says: but not probably there. So
MattGPT says: of that.
MattGPT says: Okay, so one row over here.
MattGPT says: one road wine.
MattGPT says: 12 rows.
MattGPT says: Okay, so that's it.
MattGPT says: And so that's how we set our field.
MattGPT says: Say, well, what are we gonna do? We're gonna take our needle data, frame 5. We're gonna melt it.
MattGPT says: Hold the Id column so we could select those vertical columns as the identifiers.
MattGPT says: That phone clock over there.
MattGPT says: Then the value variable again. Come from the month columns.
MattGPT says: Okay? And the variable year name. We're going to call it month.
MattGPT says: and that will automatically come from the common nation.
MattGPT says: So let's just see what happened when we've done that. Okay, so we started off with.
MattGPT says: well, Mountain
MattGPT says: 746 rows and 16 columns. That was 4 identifiers and 12 month columns. January through December. Okay. So in the long form, we're expecting a lot more, because if we're modifying it by 12 for each row.
MattGPT says: Okay? So you see, now that we've gone to
MattGPT says: 20,962 roads for just 6 columns.
MattGPT says: But let's take a peek at what the long form looks like.
MattGPT says: You can see that with that.
MattGPT says: basically see that each row represents 100 in one year and one month.
MattGPT says: Then we have a month column we got in that came from the
MattGPT says: the value comes from the column in the Y,
MattGPT says: and the reported cases is the values that were in that. So you can see that the Angola in 2011 in January it was 17
MattGPT says: recorded cases.
MattGPT says: That's this value right here. Okay. Now, if we jump down the way it's done it, it's kind of gone all the way for January first, for a country
MattGPT says: January in 2,012, there were 373 cases in Angola. And so
MattGPT says: yeah, we can see that there.
MattGPT says: Okay, does that make sense? Anyone have any questions on that?
MattGPT says: So going from wide to long.
MattGPT says: there is a way to use pivot.
MattGPT says: go back from long to wait. Sometimes.
MattGPT says: most often. Actually, you tend to be going wide.
MattGPT says: and then you just have the 2 versions, and then you can use those words.
MattGPT says: Now, when we have long data,
MattGPT says: then we can, we can see like for each country.
MattGPT says: Okay, we're expecting there's 9 years of data. So in the white form there were 9 rows of each country. But now that should be 9 by 12,
MattGPT says: because we've got one for each country in each year and each month.
MattGPT says: So you can see in the Y form, which has got 9,
MattGPT says: and we group by country. We run the same thing on the long. Then we've got a hundred names each because we multiply that by 12.
MattGPT says: And so that just clarifies what's happened there?
MattGPT says:  okay. So then, this makes when what, where the long form is is easier, is for doing things like calculating the mean or across a year, or calculating the sum.
MattGPT says: Okay? So here we can create a. So
MattGPT says: what we're doing here, as you can see that we're saying they got long form. And that's group it by
MattGPT says: the first 4 columns. Okay? So we put in all of the Id columns.
MattGPT says: That's just those are all going to be the same. But every
MattGPT says: country in every year. Okay? And then what we want to do is take the values that we're going to get in this for each group
MattGPT says: of a country in a year. We're going to get 12 roads.
MattGPT says: We just want to sum up the values for those roll roads.
MattGPT says: Okay. So instead of getting a month by month. This is the total
MattGPT says: for 2012, 2011. But Afghanistan, this is the total for 2012 for Afghanistan.
MattGPT says: Thanks.
MattGPT says: So remember, this is our pattern, start with a data frame, think about how you want to subset it.
MattGPT says: group it by road. So here we want to go country by country in a specific year.
MattGPT says: Okay? So all the rows that have the value of Afghanistan in 2011 group, those together pull out the values for reported cases
MattGPT says: and then add them up.
MattGPT says: So that would give us
MattGPT says:  measles by ear.
MattGPT says: So that makes us back to the same
MattGPT says: number. That last class.
MattGPT says: Okay, so it's up to 9 rows of the country.
MattGPT says: Okay? So melt is an important thing. When you've got why data, you come up with data and you look at it. And you're like, Oh, this is in y form. There'll be some data analysis that you can do easily with that.
MattGPT says: Okay, but you often, often your question will need you to pass the moment from why so long?
MattGPT says: So that's easy in the clock, and so that it's easier to do things like grouping and summing across years.
MattGPT says: because the equivalent of trying to
MattGPT says: do a sum year by year is that you'd have to do
MattGPT says: so. That's January. So that February, so that March, April all the way through. and then add them all together.
MattGPT says: So you get out of total column, and there is a sum function that you can do across columns.
MattGPT says: But it's not a straightforward. It's just using the group. Right?
MattGPT says: Okay? So
MattGPT says: another thing that you may sometimes
MattGPT says: do in this case. And we've seen how we do
MattGPT says:  subseting right by using filters. So let's get a list of all the distinct years in this data set. You can see the the 9 years and 2011 through 29.
MattGPT says: So
MattGPT says: it may be that you're like, actually, I want a date both for my questions and for my data analysis. I don't want to keep the whole data frame that I want to subset it into
MattGPT says: one data frame per year. We've done this quite a bit. Some of the questions that we've been working with to do things like, take this data frame and did get me this 90 ninth or 90, sixth or ninth or 2,015 or whatever.
MattGPT says: And we've done that by using a this filtering sort of app. Okay, so we've got our measles by year data frame. we select gear variable. And then we say.
MattGPT says: apply a buildup. So in this case, what we're doing here is that we're just combining our pandas and our data frame
MattGPT says: analysis. We're back in python basics. When we look at dictionaries. Remember, dictionary is an unordered set, a key of value pairs.
MattGPT says: What we're gonna do here create an empty dictionary. Okay, so we're gonna have a dictionary where the values are going to be data frames
MattGPT says: and the keys are going to be those good years.
MattGPT says: $10 a year. So we're gonna have a key that is 2011. And this associated value will be a data frame which is just the rows in this specific data frame that have
MattGPT says: this year 2011 and then 25. And so on.
MattGPT says: Okay. so you you don't often do this. But I just wanted to show you how you can combine
MattGPT says: something like a dictionary with a standard data frame. So we've gone from having one data frame.
MattGPT says: Okay, where we've had
MattGPT says: one road per country per year from 2,011 through 2019. Now we've got a dictionary where the keys are the years. And if we wanted just to access 2011,
MattGPT says: then we've got our 2011 data frame here.
MattGPT says: and it's organizing that expense.
MattGPT says: So this is a if you do get to a stage where you know you're gonna do something like
MattGPT says: me hose.
MattGPT says:  do that 9 times.
MattGPT says:  by 19. Okay, that is again where you should be going. Oh, I'm doing the same thing over and over again, and all I'm doing is changing the name.
MattGPT says: This is where you want to actually use. A loop in this case and create a different data structure where it's a dictionary which holds your subset of data frames.
MattGPT says: Okay,
MattGPT says: And actually, we could do this, you could do this. This is another way of doing.
MattGPT says: Okay, give me just the data frame the 2018. So that's quite a useful little data structure to have
MattGPT says: a dictionary where the key gets you to the subset where you're particularly something like years.
MattGPT says:  we've used. Pd, dot read Csv a lot to load the data frame in when we have Csv part of sometimes part of the step where you'll go into investigating data
MattGPT says: cleaning it. You say, like.
MattGPT says: Okay, here's the data that was able to download. It's got 125 columns. I'm only interested in a subset of those which is 20. It's got 15 years of data, and I'm only interested in 5 years. Okay, so you you can do that. You can load the data frame in. And as you can do that, subset
MattGPT says: in your notebook. But it may be that you like the subsequent sets. I don't want to. Every time I want to use these data.
MattGPT says: take out the 100 columns I'm not interested in. Drop the rows that I'm not using for the years I'm not using. So you could do that once in your investigation and clean up notebook and then
MattGPT says: you can use 2 Csv function, which does a reverse in a file. 2 C Cs will take a data frame. See that. Why, that's why it comes off the data frame yet. Y, the year data frame. So you don't to underscore Csv, give it a file name
MattGPT says: that will write it out somewhere in your folder here. So we're putting it into the data folder here
MattGPT says: so you can see what we've done here. We've started off with one.
MattGPT says: When we loaded this in. we started off with one.
MattGPT says:  there's a lot going on. And yeah, but we started up with this and measles data frame.  now we've got. You can see here
MattGPT says: a whole series of
MattGPT says: Csv files, one for each year.
MattGPT says: Okay, so that's that's an again. That's a useful way. This was the original one
MattGPT says: means of cases by country by month. Cse. We've loaded it in. We just in this case what we did was transfer it from white to long.
MattGPT says: We then
MattGPT says: Some across there across the 12 months to give us a total for the year, but then we want to separate that into year. So each of these Csv has one year where we have
MattGPT says: a rogue country in that years.
MattGPT says: Okay? And so you can see that's really easy to do
MattGPT says: from a data frame to that. So you'll often want to use that
MattGPT says: and then we can do the offset sometimes is that we come across data which is in smaller chunks.
MattGPT says: Okay? So maybe from where we download it, we've got
MattGPT says:  quite crashes.
MattGPT says: In 2,011 and 24 in 2411 dot Csv. By 2,012 dot Csv, and so on.
MattGPT says:  and actually, we want to work with that. It's just a single
MattGPT says: data rank.
MattGPT says: Okay? So we can do the same thing. So we could load each one could sort of like 5 crash. Df, equals. Pd read, Csv load or this is where we may want to use what we know already about using a loop. So if you have a list of years like we do here.
MattGPT says: What we can do is that we can create a list of data terms. Okay, so again, remember your basic type that we covered. This is why we did it. So that kind of usable overplace.
MattGPT says: What are we trying to do? We stop.
MattGPT says: We've got 9 different files here.
MattGPT says: and they have data in for the years we want to walk through each year, load the Csv file into a data frame, put that into a list and then do the next one and the next one and the next one to those years.
MattGPT says: Okay. so that's what we're doing here. Start with an empty list
MattGPT says: loop through our list of years for year and years. First, that'll be 2011, and 2012, and 2013 all the way through to 2019,
MattGPT says: then read in to read Csv.
MattGPT says:  looking for data, slash world news on the soil and then filling in that curve, breaks at the template with the current year. Value the first time all the world measles 2011. Csv.
MattGPT says: and then add that data frame into our list. Okay? Because remember, a list is just an ordered sequence of objects. So an object can be everything, anything and everything in pandas. A data frame is an object.
MattGPT says: So we're just using what we already know about piece of pipeline. But we're looping through and processing that. And then what we have. We have a list
MattGPT says: 9 objects. Each object is a data frame for a year.
MattGPT says: But really, we want to work with that as a single dateline. Okay, we don't want to have to cover this where we say.
MattGPT says: data, square brackets. 0, that gives me the 2011 data frame data, square brackets, one that gives me the 2012. I want them all stacked together. There's a very nice function, and padders called pcap, which will concatenate a list of data frames.
MattGPT says: and it will just sat them on top of each other and give you a single data range. Okay? So if we run that and we say in pat data, and the data is a list of 9 data frames, a single data frame.
MattGPT says: So we're back to where we started.
MattGPT says: Okay, so
MattGPT says: there we go, then. So we're back to 17146
MattGPT says: with violence.
MattGPT says: Remember what that looks like.
MattGPT says: Okay, so
MattGPT says: yeah, big deal. We're about to ask for. Free country columns of the year, and then the total number of visas for that country in that year.
MattGPT says: and we just wanted to double check. We have all of our years in there.
MattGPT says:  we need value. And you can see that we
MattGPT says: mostly have overlooked use
MattGPT says: okay, any questions on that and at least a basic sort of like data cleaning data, handling sort of thing. You have a data frame. You load it in your understanding about. Oh.
MattGPT says: but the way that this data and this you'll often come across this will come up widely. Data that gets published often.
MattGPT says: particularly open data has come from some sort of government body organization, or some nonprofit or something where they have been recording the data that they affected. So when you do a report, it's often sort of in some Pdf or in an Excel part, and it's for human consumption in that format. So it's going to be probably in a wide state.
MattGPT says: because a white paper was much easier for a person who might need it to look at this
MattGPT says: one column for each month in a year, and I've got to look across 12 rules.
MattGPT says:  But if I needed to work with doing data analysis on the long form.
MattGPT says: so going from way for long is the key thing. And then being able to sustain your data in chunks. Sometimes it's really useful.
MattGPT says: But it's really huge data, right? You may not want to say, Okay, I'm gonna separate this into by year or by category or some other variable and just apply a filter to get the data frame.
MattGPT says: Then for subsequent analysis. You can write that out through a separate Csv file in each one. So you've seen that. And then sometimes you want to do the reverse.
MattGPT says: Okay,
MattGPT says: So then the
MattGPT says: and we'll come back. We'll talk about ge, ge ge geolocated data a little bit more.
MattGPT says:  and I've used it in a few examples of the books and mentioned it.
MattGPT says:  if you come across when you're looking at open data, something which is called a Geojson file. Okay? That is going to be data, which is
MattGPT says: gonna still be tabular data. So each row means something. And then you have a whole series of problems.
MattGPT says: but it's also gonna have some geographical or geospatial information in it. So in the case of the shooting incidents, that would be the the location of which occurred, which is usually recorded at the point, longitude and latitude. Right?
MattGPT says: You have data which is about
MattGPT says: state at the state level. You may actually have geographic information that provides the outline of the state
MattGPT says: or little bit small enough to be Ca, account level? Or is it so? Okay? So when you see a gog somehow, and you know there's some geographical information in that geospatial one of 3 possible things that could be a point
MattGPT says: where this shooting occurred where this crash occurred.
MattGPT says: where the store is located. etc. Right? That's point data.
MattGPT says: You can also have line data which is either like between A and B, or it could be between A BC meaning select, you get line data or a route. And if you go to Google Maps and you get directions, you'll get line data, which is, where do you start? And where do you want to get to?
MattGPT says: And it will basically be a line, a whole series of lines between those things. So that's the line data.
MattGPT says: you you that's the least common income, probably. And then the third type is shape or polygon data, right? So that's a whole series of points, which are there
MattGPT says: defined as a series of fine, but they
MattGPT says: joining together the main shape and that shape to find something that could be stable
MattGPT says: could be. Yeah, on the country could be out lying on the street.
MattGPT says: alright, so that's that's a polygon. Right?
MattGPT says: So here we're loading in some Geojson data for the countries in the world. Okay,
MattGPT says: and you'll notice that we're doing it slightly differently because we're using this.
MattGPT says: Not a lot of people. Geopatus.
MattGPT says: If the geopanda is bearing a pandas plus geospatial or geographical program. Okay?
MattGPT says: So
MattGPT says: just as we've done with pandas. We don't want to be typing pandas. Dot read underscore Dsd. Or Pandas dot mount or merge, or whatever we use. An alias a Pd, and that's the standard sort of alias. The Geopandas, is just geop sorry. But geopandas.
MattGPT says: Okay. So the way that we load in a Geo. Data frame is with GPE. Dot v underscoref file.
MattGPT says: and so
MattGPT says: let me do it by that. But then we can treat it in the same way. It's got a number of rows, and it's got a number of cops. Okay? So we've got 255 rows.
MattGPT says: We've got 4 columns.
MattGPT says: What does each row represent? Well, let's take a peek at that. and you can see that what we get. This is a really simple data frame. Each road represents a country
MattGPT says: admin is a country name Iso. International Standard. A. 3 is a free letter abbreviation for all the countries in the world. There's also an a 2 version of that which is 2 letters.
MattGPT says: There's a whole bunch of standard ways of
MattGPT says: the world, have an organization, have their own. The World Bank has their own they tend to be very similar. But there's some businesses that
MattGPT says: for those that just identify what Metro is, and then you can see that we've got something called a multicolon under the geometry. Okay? So a geopanda's data frame the thing that makes it a geopanda data frame that magic
MattGPT says: thanks, please. And the geometry part. Okay? So
MattGPT says: we have. Here is a whole series, pairs of points, latitude, longitudes that's separated by call it's at each point in a polygon
MattGPT says: find in that way. So we'll either be just a polygon, or you'll see sometimes a body polygon. That's where you've got multiple shapes defined.
MattGPT says: That may not be. That may not be the tickets.
MattGPT says: Thanks. So the cool thing about geopand is is that when you have
MattGPT says: a Geo data frame, and you pass. You say plot.
MattGPT says: It knows. Oh, this is Geo. This is shaping up.
MattGPT says: and so I'll just use that map. But I'll treat the x-axis, the value and the wires in the box.
MattGPT says: Pretty ugly the defaults! How it comes out! It uses a
MattGPT says: it uses blue to fill in all the colors. So you can. You can reset those colors. The color is actually the fill color.
MattGPT says: and then the edge, however.
MattGPT says: is the outline of the shapes.
MattGPT says: And then you can resize that a little bit or an aspect ratio.
MattGPT says: So it's right.
MattGPT says: Actually, see here.
MattGPT says: map of my countries in the world.
MattGPT says: Okay, and you can see that it's right down pretty fine. But I don't know. Everything is fine again. Right? And so, Jason, so that's the mobile phone doing
MattGPT says:  But each row is a country. And then the geometry just was, how do you draw that country
MattGPT says: using each of requests? So it's a really nice sort of
MattGPT says: way in which you can easily just expand. You can add geographical mapping and geographical analysis.
MattGPT says:  to your data.
MattGPT says: So we'll talk a little bit more about geopanters. I'll give you some more examples how you use this. Some of your data projects may have Geo
MattGPT says: spatial data, and we can spend some time. Make sure you know how to handle that.
MattGPT says: We've seen a few examples of this. sometimes we have 2 data frames. Okay, and
MattGPT says: that you want to
MattGPT says: combined together. So we saw concadination, which is like, you've got a whole bunch of subsets of. You've got a whole bunch of data frames, but they've all got the same column to them. And so you just
MattGPT says: okay. But merging is where you may have 2 data frames.
MattGPT says: We have at least one common problem. Okay? So let's say, we have a data frame which has
MattGPT says: identify person, one person, 2,
MattGPT says: button, free and bus and 100 that's able. Okay? And then we've got H
MattGPT says: from there. Okay? And that's in one day to frame. Then let's say, we also got
MattGPT says: what
MattGPT says: another data frame, which also has the same identifier. And so we've got information about person one. 100.
MattGPT says: Okay. And then we've actually got there.
MattGPT says: favorite column.
MattGPT says: And from state web, you know, got a whole bunch of variables. But right.
MattGPT says: what we want to do is that we want to merge these 2 together so that we match this
MattGPT says: road and this row here.
MattGPT says: so that we end up with a data frame that has the Id and the age.
MattGPT says: and the favorite colours
MattGPT says: and the home state. We've got maps on on
MattGPT says: now, if you're identical right in terms of they, you have 100 rows, and they're ordered in that way.
MattGPT says: and you can think about how you might do this in excel or in Google sheets same way, and then copy these 2 columns here, come here and paste them right here.
MattGPT says: That will work. But you have to be absolutely sure that you've got all of the same identifiers here.
MattGPT says: same number, and that they're in the same order.
MattGPT says: But sometimes it may be that we only have
MattGPT says: this data about
MattGPT says: 500, yeah. But we still wanna bring where we have information that's added into this later, right here.
MattGPT says: Okay, so that is where you'd use the merge. And so merge is where you have at least one common column. The values are common that certainly have the same names.
MattGPT says: And what you do is say, merge.
MattGPT says: Okay. So we say, Pd merge. And so then we've got 4 things that we need when we do a merge data frame the right data frame. And then we've got the column which we're going to use in the left data frame, and the column will get us in the right data frame.
MattGPT says: And so this is what I merge the place.
MattGPT says: So we've just loaded in that Geo.
MattGPT says: Json file which had the countries in. Okay. And remember here, like, we've got an Iso
MattGPT says: free, which was our three-letter abbreviation for a country.
MattGPT says: If you look back at the measles data.
MattGPT says: then we had this iso 3 con, which is the three-letter family code to each of the countries in the measles here. What we want to do, what we're trying to achieve here is that
MattGPT says: we want to have this database here
MattGPT says: with the country name and a the shape of the country. But we want to have a number of cases of measles in a particular year as an additional problem.
MattGPT says: And so, and we're going to do it for 2011.
MattGPT says: So we're going to take our 2011 data frames.
MattGPT says: Then in notice the different names of the columns of the values. The same. It's called Iso underscore, a free in the Geo in the world. Gdf data frame. But it's called Iso free in the year data frame.
MattGPT says: But if we merge these 2 together.
MattGPT says: you can see that now we've got 9 columns. and that
MattGPT says: so what does that look like? Well, let's take our merge data frame.
MattGPT says: Okay? So you can see what it's done.
MattGPT says: Remember, in our year, when we had, we had 5 pounds we had is a free region countries year. And then the reported cases.
MattGPT says: And what it's done is it's mapped those up so, Andorra.
MattGPT says: it's matched with Andorra here and choose this common column here? A and D, and it's matched with A and D,
MattGPT says: okay, so it could be in a completely different order in the, in the in the data frame. When you do the it's matching on value, not on the decision.
MattGPT says: You have a different number of rows in this data frame. And this data frame.
MattGPT says: Long as they have some common values, they'll get merged together.
MattGPT says: Okay, so you can see. Now, this is cool, because what we have is that we have Geo data here. We have the shape of the country, but we also have some numeric data.
MattGPT says: the number of cases of measles
MattGPT says: specific year for that.
MattGPT says: So we can.
MattGPT says: just expand our plot
MattGPT says: arguments here. And so, instead of specifying the color, fill the shapes and the edge color, we just give it a column from our new Geo data frame and say, use the reported cases values as a way.
MattGPT says: it's pale
MattGPT says: each of the countries.
MattGPT says: Okay? And so this is called A, for like a heat map, a country heat map. Okay? But then over here you can see the color bar. This is the greatest color sort of mapping here. But you can see that this goes from blue up to yellow through green.
MattGPT says: Okay? And so lowest values are down the bottom, and they're dark blue and highest values.
MattGPT says: Are in yellow here. So we can see the countries which have the highest values of measles across the world in 20 other.
MattGPT says: And yeah, how do you change this
MattGPT says: scales? That? Yeah. So there's the arguments for that good. Yes, let me show you a company. So you may think, yeah, that's really ugly right here. So
MattGPT says: and again, you can use a C map, which stands for color map. Okay? And I'm putting in a jump value here as a kind of app to show you how to use this, you'll see down the bottom here. This is the list of and you didn't Google this. But on that there are things called color palettes.
MattGPT says: And so you you can make sense of some of these you got things like
MattGPT says:  YLG of GN, and that's yellow degree you'll see 3 parts that yellow, orange brown, then white, elbow out, and so on. And then you got underscore.
MattGPT says: is whether it goes from dark to light or light to dark. Okay, so let's just say we pick one of these?
MattGPT says: I always like the oranges, but don't read them.
MattGPT says: Okay. So if we set up
MattGPT says: key mappers. oranges.
MattGPT says: then, you see, now we've got light orange. because the white is low and dark is high.
MattGPT says: If we specify that as
MattGPT says: underscore R, then that reverses it. so we'll go from stop being low.
MattGPT says: so you can play around with those. Those are the colour maps. Okay?
MattGPT says:  And then to answer your question, Max. If there's
MattGPT says:  there's an argument called legend keyword, or A. W.
MattGPT says: Yes, okay. And this, there's a whole bunch of ways in which you can control that that legend of the size that color. Okay? And it's so take the dictionary.
MattGPT says:  and one of the things that you can do is that you can shrink the size of the dictionary
MattGPT says: by proportion
MattGPT says: between 0 and one. But it would say, like, Let's make it
MattGPT says: 40% smaller.
MattGPT says: and then see that you guys are on that.
MattGPT says: the
MattGPT says: maybe the
MattGPT says: like to dock. It's better.
MattGPT says: Yeah, something on that.
MattGPT says: And I've probably made this too big.
MattGPT says: Okay.
MattGPT says: so this is really this fun stuff to do right. You can see how this can be stuff to go towards a really interesting exploration of data. But then also for explanation. If you're wanting to in a data story.
MattGPT says: in a blog post or something like that, you've got geographical data.
MattGPT says: the context, the big picture here
MattGPT says: you need. If you find a Geojson file which has the units that you want to plot onto right? And in this case it's countries in the world. We found that one.
MattGPT says: That's just a data frame. So it's a table. But it's got this special column, the geometry column, which defines the shape
MattGPT says: that's useful for just from applying those sheets.
MattGPT says: But really, what we want to do is get one or more
MattGPT says: numeric pieces of data in there.
MattGPT says: So that we can then follow it. Watching. Sorry we can name
MattGPT says: build the shapes by some variable in this way, Kit. So we're here. We're seeing like
MattGPT says: these are the places where there's larger instances of measles.
MattGPT says: and then we can sort of
MattGPT says: we could do. We could do a fully merged version of this now, so that we go back to our data frame. That was all of the years together, all 9 years together.
MattGPT says: we can merge those 2 together. Okay.
MattGPT says:  And so then, what we end up with.
MattGPT says: you can see here the data frame that you put first.
MattGPT says: So we flip the order here. We put measles by year that comes
MattGPT says: first in the merge version. So we get our. But we get our 5 columns from our needle data rank 1, 2, 3, 4, 5, and then we get our columns from our
MattGPT says: well, gee, Gds hypocrite!
MattGPT says: The order that you put them in
MattGPT says: is important, and it also affects sort of emerging there. So we actually want to do this the other way around. Okay.
MattGPT says:  And you can see
MattGPT says: now, what we have
MattGPT says: is that we've got 9 distinct years in there.
MattGPT says: And so for each
MattGPT says: country in each year we've got a number of reported cases, and then we've got the shape for that country. And then there's going to be the same country again. So we're getting repetition and the shape here. Thank you. It's just another column.
MattGPT says: What's it gonna have in there. So this means that then we can do
MattGPT says:  We could do something like this. So how do we display it year by year?
MattGPT says: But now, what we're going to do is 9 separate plots one per year. Okay, so we're gonna
MattGPT says: bloom. Fruit means year
MattGPT says: filter out just the roads for that year in our mode data frame.
MattGPT says: create a book
MattGPT says: for the
MattGPT says: the countries are not yet.
MattGPT says: Add it up a little bit. We want to keep the same
MattGPT says: be made of B. Max.
MattGPT says: So year by year you're gonna have a different number of minimum number of cases and measles, a maximum number of cases.
MattGPT says: So what we calculated first was like the total number of cases.
MattGPT says:  across the largest number of cases anywhere in those 9 years. Okay.
MattGPT says: but this that gives us a year by year.
MattGPT says: way of seeing how does time develop? I mean, how does the measles cases develop, drop and
MattGPT says: decrease? And so you can see the key thing that's really important when you do. This kind of visualization
MattGPT says: is to make sure that the color bar has exactly the same minimum maximum value. By yeah, why, you can't keep.
MattGPT says: We didn't set B Min. And the Max.
MattGPT says: Then we'd have a different set of the colors would mean different things in each year.
MattGPT says: Okay, cool any questions on that.
MattGPT says: I love this Geo stuff. I think it's like super exciting.
MattGPT says: There is a way in which you can make this interactive. Maybe I'll show you that on Thursday.
MattGPT says: where, instead of having them all stuck on top of each other, you can add in a little slider that will go from 2011 up to 2019. And you could. because they're just creating the right function to do that.
MattGPT says: Okay.
MattGPT says: Then, number 4. I'm not going to talk about today and
MattGPT says: just ask you if you take a look over it between now and Thursday. Just repeat some of what
MattGPT says: we were just talking about with merging, with some upbringing on another data set.
MattGPT says: And so whatever to load.
MattGPT says: files.
MattGPT says: Yeah. So take a look at this notebook. Just gives you another data. Set some examples.
MattGPT says: Okay,
MattGPT says: the last thing I want to just quickly cover today.
MattGPT says:  is telling stories with data. And so
MattGPT says:  kind of like this manager of my favorite. So
MattGPT says: from an article about David storytelling.
MattGPT says: okay? And so what happens? A lot I see is that
MattGPT says: when people think about telling stories of data doing data, science sort of data, journalists and stuff tend to go to the right and top right hand. The visuals can be really interesting. But it doesn't. Really. You know, people think like, if I can understand my data presenting in such a way that it's like in a dash to the full, you know, change. So that's different values.
MattGPT says: That gives people the most flexibility. They can look at the data themselves. And they can figure out what's going on.
MattGPT says: So that's not really data storytelling. Okay, that's some data analysis and some data, preparation and data presentation.
MattGPT says: But it's not telling a story from data. And just you know what we just did where we plotted the measles cases per year. That's not data story product, either. Right? There is a probably a story in there.
MattGPT says: But it needs some narrative around it. And the narrative actually is the most important thing, because people
MattGPT says: still have a preference for. And we'll always have a preference for words and for so
MattGPT says: for all the structure in a linear way, a story and sorry the foundation of what we, how we communicate.
MattGPT says: You know those years. We have people tell us stories, and then we learn to tell stories as children.
MattGPT says: and that kind of fundamental how we communicate when we meet. So oh, you know, guess what it's fun to me right when you're even think about. You know you may not notice it, but if you actually
MattGPT says: listen to some of your friends talking, you're part of that you'll see that storytelling is like foundational to most of how we can be here. It was all just happened to me from a narrative structure, you know. Say, oh, once upon a time, you know, you say, Hey, you know, guess what happened earlier, that
MattGPT says: okay, so narrative is actually the key kind of part of that. And then when you put that together with
MattGPT says: understanding data processing data looking for patterns, all of these things that we've been talking about.
MattGPT says: and then, appropriately thinking about how you use the right visuals. Bring those 3 things together. Then you have data story time. Okay,
MattGPT says: And so
MattGPT says: I talk about this a lot more in my data stories from data class where I go into this in more depth. But I want to give you the same here again. It's a nice article from a number of years back. David, journalist com.
MattGPT says: where it talks about data stories and and gives an 8 part
MattGPT says: sort of typology, of ways to think about and to use stories. To use data in stories. Okay? So I'll give you each of these. There's free notebooks I can. I'll give you those as sort of reference as well.
MattGPT says:  But here, what I wanted to.
MattGPT says: I like Vanessa.
MattGPT says: So this these notebooks use these 3 different sorry, these 5 different data frames.
MattGPT says: I'm not gonna
MattGPT says: talk through all of these. But these fit and what we've seen before load the data, how many rows, how many columns? What do we throw represent? What are the columns
MattGPT says: describing those? Okay?
MattGPT says: for these 5 data sets? And then making some observations. So this is all part of that first step of exploratory data analysis
MattGPT says: and making observations.
MattGPT says:  and so then the 4, the first 4 types of measurement.
MattGPT says: in the article. Sorry of of
MattGPT says: type. The first 4 ways of using data in stories are measurement, proportion, internal and external comparison.
MattGPT says: Thank you.
MattGPT says: So measurement is like the simplest way in which you can use data. Basically, you've got your data
MattGPT says: measurement would be telling somebody the number of cases of measles in Angola, in 2,011 in January, for instance, that was just the number of things. It was like 17 or something 117, something like that.
MattGPT says: Okay, so and and often, that's the most important or the most interesting thing there is actually just to give.
MattGPT says: you know, a single measure of some kind. So this will be a count will be a total could be a percentage
MattGPT says: but usually it's gonna be an hour or 2 or
MattGPT says: okay. So the way that I want you to think about this, the next step of moving on your exploratory data analysis. And you understand the data, what are the key variables? Then you're meeting all the time is, what questions can I ask of these data? What questions can I answer? Give us answers? Okay, so you should be like curious, curious. Oh, this is interesting. This is interesting. Okay? So I hope
MattGPT says: I know, you know, one of the key things being good data story teller, good data analyst, a good data scientist, whatever you wanna call it, is a curiosity that says like, why is it they report to me, why is it someone else? What can I do with it? Right, you just get like Super, giddy and excited, like a little kid, you know. It's like, Oh, I wanna take, I wonder if that. And as you're looking at data, so that's interesting. Oh, I wonder what I can make of that. Okay? So
MattGPT says: you know, I'm sure we've all
MattGPT says: been around
MattGPT says: 5, 6, 7, 8 year olds who are at that stage of life. Why, this, what? That you know why? When, where, why is this? Why, why, right? Everyone knows kind of what that's like. Right?
MattGPT says: Try and take on that attitude when you're you're working with data. Okay? But you're just constantly thinking like, Oh, why or wonder what is going on here?
MattGPT says: Okay? So always ask some questions with, Wake up.
MattGPT says:  And
MattGPT says: yeah, and yeah, try and pick a childlike sort of curiosity do so
MattGPT says: and so what we can do with these things is that you know, and what I want you to do as the next stage and this will be, I'll put us in. Come up with at least 6 questions that you can ask
MattGPT says: traffic accident later in Philadelphia.
MattGPT says: You've looked at it, and you figure out what the whole bunch of variables, but it's got like each incident when it happened where it happened who was involved. And then this type of vehicle.
MattGPT says: So so one of those is 5 people involved. Yes, no, or I think it was. And so you can ask the question, how many both have accidents reported in Philadelphia? 17 and 4 bicycles?
MattGPT says: Right? A good question, just a measurement. How many? And then the other data set that we had in Philadelphia Zip Post.
MattGPT says: and it's measurement of the amount of lead in a blood sample, the children taking over a period of 201-32-0150kay, so could just be, I mean, I'm curious in this data, like,
MattGPT says: how many children are actually screened.
MattGPT says: and you've known you can see that there's a count for Zip, so you could after that.
MattGPT says: right?
MattGPT says: The case of the measles that we just were looking at. We saw that we got the country by year, so a really simple question and shift like, oh, in a country pool in a year 2015.
MattGPT says: What were the total number of cases?
MattGPT says: Okay. so once you understand your data from doing the exploratory data analysis, next step is like. And and and this doesn't need to be a separate step. So I do the exploration, and then I'll do the questions.
MattGPT says: They're naturally going to happen at the same time as you're looking at the data you're like, Oh, wonder this. I wonder that? Okay, so always generating questions. But after the the exploration, then you group together those questions and say, Okay, so what are the key questions that I want to answer here?
MattGPT says: Free questions from freedom data sets that we're looking at here that this is a structure. For how you go about answering those questions.
MattGPT says: Okay, so in your assignment, mark, you, follow this structure and to lay it out in this way have a mark down cell
MattGPT says: sets out what your question is in this case, how many robotic accents reported in the Philadelphia area above bicycles.
MattGPT says: then in Markdown form, before you write any code, what are going to be the steps I'm going to need to take
MattGPT says: to answer that question. Thanks. Now, you understand your data because you've done your exploratory data analysis. Okay? So in this, each row represents a
MattGPT says: a crash incident.
MattGPT says: So I need in row by row. I need to select specific rows. And my question talks about this area so that don't need that part of the year is specific. You have a year built up.
MattGPT says: and then you need to have a filter that gets me roads where
MattGPT says: there is bicycle involved that variable that you've you've announced, and you've you've explored is the bicycle count.
MattGPT says: Okay? So
MattGPT says: 2 filters, one to get crash year equals 2017 and another to get bicycle count greater than 0. Then when you put those 2 filters, we're gonna combine them to create a subset data frame, we're gonna pull by press 2017,
MattGPT says: which is Jeff rows the 2013 where bicycle count is greater than one. And then to answer the questions, all we have to do is just count up the number of rows.
MattGPT says: Okay? So then, once we've understood the steps that we go through and we, we are able to put them out. And
MattGPT says: you know, in a series of steps like this, then that's really easy to translate into the code right? And we talked about it a little bit before
MattGPT says: generative AI, cool tap, Dp.
MattGPT says: co-pilot, etc., etc., and become more more prevalent. One of the things that really good at is helping you write code. Okay? But it should be just that that code of systems.
MattGPT says: They will only give you as good a alpha as you give them. Good. Input. Okay, so being able to break down and understand your data and break it down into steps like this
MattGPT says: is going to be a good way to use those tools as in the system that you're going to hand.
MattGPT says: We know enough pandas and a pipeline to be able to do it without one. But like.
MattGPT says: there's no reason why you shouldn't. So. But yeah, once we understand that, how do we create filters?
MattGPT says: We're from our data frame.
MattGPT says: We're looking for Rose 2017. That's up in 2017, builder, and then a bicycle bicycle count is a new, merry, variable. So then, we say, greater than 0 one bicycle involved, 2 up to the however, many builders
MattGPT says: we can run those.
MattGPT says: and then remember what a filter is. It's just the same number of rows as you have in the data frame that you filter.
MattGPT says: you've got less mature false values. Suppose means
MattGPT says: third row don't have 2,017 in the past year. Okay. we have those 2 filters.
MattGPT says: We just create a subset by combining the 2 filters. So crash underscore yet square brackets. Then you can buy one or more
MattGPT says: row filters in combination. Okay, so the Ampersand there is a logical, and so it must be both true for 2017 and true, for by count greater
MattGPT says: 0.
MattGPT says: Those both are true. Then select that bow, and that's going to be yours.
MattGPT says: Go to the data frame, and we're going to pull that by past 27.
MattGPT says: Okay, so then. by press 2017, we've got a lot more rows, a lot more columns in there. But if we found each of those you can see. Yeah, crash year is here.
MattGPT says: We found the
MattGPT says: But let's just buckle until week. One.
MattGPT says: Which is what? Yeah.
MattGPT says: let's pronounce
MattGPT says: question.
MattGPT says: I mean twice in.
MattGPT says: Hmm.
MattGPT says: okay? So you can see. Yes, every row that comes up in a sail frame is 2017, and does have a white amount greater than 0. Right?
MattGPT says: So filtering was good. Set free. If they're just half the number of rows.
MattGPT says: Okay? And remember where we've got our sub data shape, which gives us the number of rows and the number of columns to take the first of those.
MattGPT says: and then you can see that we've got an answer. There were 430 crashes involving bicycles in 2,017,
MattGPT says: and then the last step is to put in a mark down cell.
MattGPT says: And so write that answer out. okay, so that's the structure that I want you to use
MattGPT says: for the third part of the assignment. Okay, I know we'll talk about again on Thursday. When I give you that, David, you're gonna have. You've done your data analysis.  come up with questions. Some of them may be as simple as just counting things and doing measurements.
MattGPT says: You'll see in this notebook, and there's other ways in that typology. Sometimes it will not just be
MattGPT says: which it could be a question like, in which year was there the greatest number of crashes involving bicycles across all the data that you have. So there you can do something a bit more complicated which is grouped by year.
MattGPT says: You know. First build it by by count greater than 0. Then group by year into account.
MattGPT says: or do a value count on years, and then you'll have a round table, and then you just read off the first row which year, whether there's at least number. So so there can be sort of revolt. More than one variable. Do comparisons. You may want to do, person, between 2 years, or something like that.
MattGPT says: Basic structure is the same. Think of the question set out. Question, how do I do it. What are the steps? Break it down, set layout steps in in pros and text before you trying to code.
MattGPT says: Thank you to the structure, insert the code. Get the answer. have a mark down cell. That that explains what the answer is or states what the answer is.
MattGPT says: okay. So I have a look. You can look through the rest of this notebook. There's a few more questions there.
MattGPT says: you can even try using
MattGPT says: using that structure. Okay? Any questions. Anyone.
MattGPT says: Alright great cool. Thanks, everyone. So I'll see you on Thursday.
MattGPT says:  And I said, this week you're just gonna have a quiz you on Friday. The readings on the this will be available tomorrow.
MattGPT says:  and on Thursday we'll go through the rest of this, and I'll give you the struck. I'll give you the assignment, for
MattGPT says: when we do on Sunday but a week on Sunday. Thanks. I think no one.
MattGPT says: what are you doing for video?
MattGPT says: Okay? Because of the accounts on the people? the there's a see? Your project have?
MattGPT says: Okay? And I think
MattGPT says: so far we've got, we've got that number of groups that I know about or I remember. But I have not. Everyone is in a group yet. So if you can go there and pull yourself into
MattGPT says: the group that you for
MattGPT says: there's people here who aren't yeah in a group. Then you can add yourself to.
MattGPT says: you can create a group.
MattGPT says: a. okay. Lucas. Mid eighth
MattGPT says: Pam is in that dude. Right?
MattGPT says: Groups to one.
MattGPT says: yeah, okay, yeah. Okay, so you can get that, you can do that. Now, everyone and and check that. These groups are right. You can move yourself around if they're wrong.
MattGPT says: Okay? So I think that's what I find.
MattGPT says: What I want you to do, Ben.
MattGPT says: you third on Login's campus. I think you probably use this group stuff to call. But then you go to your groups. Homepage. Okay? So like Max and Kate, let's go to your home page. Right.
MattGPT says: Katie.
MattGPT says: this is where we'll kind of use it for like planning and discussion and stuff like that. Okay, so just like regular sandwich, like you can make announcements, get an email around you and then have discussions. Okay, so what I want everyone to do just to get started in your group. Go to go to your group page.
MattGPT says: And then in the discussion, thought of Fred.
MattGPT says: which is sort of like project ideas.
MattGPT says: Okay, 2 meters. So it's not a new Fred project ideas. And then in there questions. okay, so I'm going to start this one, Pete.
MattGPT says: Okay, so he is. And then go ahead and put in what your ideas are. If you've got a really clear idea that you know exactly what you want to do, then in there kind of a couple of sentences about what project ideas.
MattGPT says: what data set you think? You're going to use?
MattGPT says: You have found them already with some URL, and then they just use that as a face kind of starting place to start to organize what it is that you're doing. Okay? So everyone got that. Go to canvas, go to your project, group home page
MattGPT says: in discussions.
MattGPT says: start any project ideas or project planning tool or whatever you want, and start using that. Okay? So let's take 15 min to sort of use the time. Yeah, I'll delete this one.
MattGPT says: Okay. So if you can spend the time talking about yourself and doing that. And then so like, here.
MattGPT says: this is so within minutes.
MattGPT says:  what the idea is?
MattGPT says: URL down.
MattGPT says: is there way?
MattGPT says: Yeah, you can also use.
MattGPT says: Yeah.
MattGPT says: underpaus, right?
MattGPT says: Desert, right?
MattGPT says: Yeah, and there are data sets.
MattGPT says: I definitely know, that's in there. Actually
MattGPT says: data as well.
MattGPT says: Right? So
MattGPT says: then maybe, like.
MattGPT says: I guess you want to try and identify.
MattGPT says: Keep from doing your time. I think real good.
MattGPT says: I mean.
MattGPT says: oh, that's
MattGPT says: there's a lot of open data.
MattGPT says: I guess.
MattGPT says: Or.
MattGPT says: yeah.
MattGPT says: that's right.
MattGPT says: And then.
MattGPT says: right, so
MattGPT says: cool, yeah.
MattGPT says: how are we doing?
MattGPT says: We had an idea about one data set
MattGPT says: level, probably
MattGPT says: but then you could have a
MattGPT says: specific places.
MattGPT says: You get, what are you trying to find out? Right?
MattGPT says: I am
MattGPT says: this code. Then there's a lot of data.
MattGPT says: Yeah, here, I would, you know, keep ideas. And then maybe if you get that, 3 or 4 of them.
MattGPT says: Okay, so
MattGPT says: yeah.
MattGPT says: so the
MattGPT says: to see you now.
MattGPT says: you could also like
MattGPT says:  If you're a group of 2, you still need at least 3 data sets right. 3 data sets is sort of the minimum to make a good story could be one core data set you've already found. And then you're thinking about what are the other data sets. You're gonna bring alongside that. Okay? But it could be that there's no like. You must have X number of data sets. But there is a minimum
MattGPT says: gotta have free, and that your project is so like.
MattGPT says: so just wonder hopefully.
MattGPT says: And then it's kind of like.
MattGPT says: okay
MattGPT says: to myself.
MattGPT says: You can figure out some correlations, too.
MattGPT says: Oh, yeah.
MattGPT says: got it.
MattGPT says: That's how I'm very messed up. I say.
MattGPT says: sure.
MattGPT says: that's what. Slightly, you know, 62%
MattGPT says: spots like real quick. Can't be sure. Yeah.
MattGPT says: we'll do some hand-drawn charts too much.
MattGPT says: Oh, no. 13%. And then we're like, Oh, wait, this seems a little too. Okay. Oh, yeah, look at this. There's a screen to the line.
MattGPT says: They have 3 to see
MattGPT says: support notion. and
MattGPT says: that's all. You get
MattGPT says: the meetings awesome.
MattGPT says: They shared. So it is.
MattGPT says: was just like one more. So
MattGPT says: I say, I know God. But
MattGPT says: you want to add to our website.
MattGPT says: There has to be some.
MattGPT says: You can even increase your market.
MattGPT says: Okay, please.
MattGPT says: Okay.
MattGPT says:  alright. But I didn't ask to talk about it.
MattGPT says: poking closer and
MattGPT says: that's just in there, just museum within.
MattGPT says: Yeah. alright.
MattGPT says: Okay, this call.
MattGPT says: look at some programs.
MattGPT says: Pennsylvania. Yeah.
MattGPT says: this is Nicole.
MattGPT says: your customers. everyone's good.
MattGPT says: This is part of what.
MattGPT says: So are we looking for?
MattGPT says: I was taking care of customize this.
MattGPT says: I think that's
MattGPT says: so. Not
MattGPT says: yeah. Okay.
MattGPT says: I didn't want to go with.
MattGPT says: So it might be worth.
MattGPT says: okay.
MattGPT says: yeah, it looks like real.
MattGPT says: I gotta jump into into the groups and I'll look at your discussion, Brett, right, and I'll give you some feedback on what we already put there, and some additional questions that I might have elegant box. We don't know how to get it out, like those are the kind of things to know there's questions to me, and I'll try and have a quick look at my give you some ideas about how it's most of the time as well.
MattGPT says: Right?
MattGPT says: When I go to the group, Tab and I go to discussions. This is what it looks like.
MattGPT says: Yeah, okay. So you should add.
MattGPT says: you don't have an add to add something and call it like predict ideas or project planning, or to then you can create multiple credit. Yeah.
MattGPT says: records of respect.
MattGPT says: Okay, so if you wanna fetch
MattGPT says: if you wanna take take a look at forward 13, right? This is a few things I wanted to show you. Some of this, I think, will be relevant what we're we're doing that. Okay,
MattGPT says: Okay? So it's all small notebooks in here. Just some preference stuff to you, basically, the first one I wanted to just pull attention to
MattGPT says: So, selecting rows, I just wanted to to mention this. Some of you may, or already have sort of
MattGPT says: sandwiches.
MattGPT says: Figure this stuff out right? But like, this was the data set that you were working with the World bank data set on science. 5. Okay? And so.
MattGPT says: Jessica. we're doing things like. create a data frame which is just. For you know, 2,014.
MattGPT says: Okay, so kind of go to way in a way that I think is is the way to stop always is the road filter. Okay? And so when you do a real filter, remember, you know. So that call off a combination of on and do quality tests, or greater than, or you know, combination of that. So we've used this kind of, we've used this approach both of the time
MattGPT says: on the stuff that we talked about before, and that's a good way to go right the final filter, and then use that filter. And the reason that you find the filter separately, and instead of having a big, long query, where you can nest these things
MattGPT says: is that you often find you want to reuse stuff like that, so give them a name, makes it easier to read. And then you can combine you 24 people and
MattGPT says: region filter, or something like that, right?
MattGPT says: For like Africa, or something like that. So that's that's using row filters.
MattGPT says: You can see when we do that. We've got one year. We've got 154 countries, and then the 11 variables.
MattGPT says: That is, in later version of the Pandora, and set the versions that you run here, and you'll see more and more examples of using this is that there is a query function. And so you can use that on a data frame. And then you give it a query string.
MattGPT says: Okay? The query chain is a little bit off. Okay, what it actually is is a python expression.
MattGPT says: And so it it.
MattGPT says: but
MattGPT says: okay, so we can just say a query. We don't need to reference the data frame, because it must be a call in there. Okay, so you can, you can play around with that you can. You could combine our things within that.
MattGPT says: That will give you the same data frame, same number of rows. Okay, so that's the second way, using query.
MattGPT says: and then the other and this came at the best of quite a bit about the confusion there was on question 15 BI think it was where
MattGPT says: there were different ways of subsetting, and most of you used weight and the filter way one and I happen to use number 3 and then realize, oh, you know, I haven't really talked about it.
MattGPT says: But when we look at our data frames that we've been working with
MattGPT says: by the quote. When you learn load it in. The index will just be a new member index from 0 going through. And that's things you get involved here. The way that you use that index we've seen is like selecting rows. You can use the lock, and then the number there, or a range of numbers, or a list of numbers. Okay, so you can select rows in that way.
MattGPT says: But you don't really need you don't very often think, Oh, I want the first 50 rows, or I want rows 170 to 132, maybe. In some instances you might, if you know kind of what they mean.
MattGPT says: But the index actually is really flexible, and you can set the index to the anything that you want. It doesn't have to be a unique number or a unique value for each row.
MattGPT says: Okay, so what you can do is that you can say, actually, let's take this year column, and we'll make that the index in the data frame.
MattGPT says: Okay, so what happens when we do that?
MattGPT says:  okay. the data frame, we say, set
MattGPT says: set index. Yeah.
MattGPT says: You can see, then what's happened is the year column and disappeared out of the data frame and become the index.
MattGPT says: Okay? So the index and the index is now the Year Index. So it's going to be, there's going to be repetition in there. So what's useful about that is that then you can use that index for various things.
MattGPT says: and one of the common things would just be to say.
MattGPT says: Look for the location of the row, and then you can put the value in here. Now, it works really nicely with here, because it's just a number. Okay, it will also work with strings. You have strings in that. But that, then, is the equivalent of creating a filter.
MattGPT says: Right? So we've now just got we just indexed by row
MattGPT says: all of the growth at 99. So that's exactly the same
MattGPT says: index that year.
MattGPT says: Okay? And so then you've lost the year cost over the years. You want to be able to find the year column. That is now the index.
MattGPT says: You may want to go on using the index, but you may just wanna then actually put it back into the sub data set. And that's when you use reset index. Okay? So if you see code that looks like this or chat Qv gets you code that has in depth to the men at least know what's going on.
MattGPT says: When you say reset index, it takes the index. then put the end of the column, and then just go back to a numerically ordered one.
MattGPT says: Okay, so that's using indexing and so we could use it for this query, get a tool for 2,014, but we set the end of the year that would select all the rows that have 2014 index, and then
MattGPT says: we just need to remember that we need to do reset.
MattGPT says: We want to get back to the data thing like that.
MattGPT says: Okay, so
MattGPT says: there's a lot more that you can do with indexing categorical variables were
MattGPT says: so obviously not gonna work with a new merit kind of chronicated one. It doesn't make any sense to set up the birth rate, because you can't do things through an in-depth like
MattGPT says: something. Query.
MattGPT says: okay, any questions on that.
MattGPT says: All right. Cool.
MattGPT says: So I want to try and quickly get through these. This may be relevant to some of your projects. Now, okay, so some of you are going to come across data which is in Pdf form.
MattGPT says: Okay, thanks to Anna, who put this particular Pdf in your data story
MattGPT says: on the sign. Or I think it was right, which was, we're doing like we had the Acs data about. So here's like something that you may come across
MattGPT says: right? This is a report a lot of times.
MattGPT says:  this is what people think like. In government organizations when they say, Oh, we're doing open data. This is the right report. And this is the data that we want
MattGPT says: and we have to figure out, okay, can we use that? You know? Sometimes you can do this and try and copy and paste it in but you know, can be gnarly but this is, you know, data that's formatted.
MattGPT says: But is there a way to get at the data in this right? So this particular report?
MattGPT says:  you know some. Sometimes you'll come across it in Pdf form, but it's just a plain table, and that's much easier to get stuff out of them all like with. But like this is a little bit tricky.
MattGPT says: And so there's free libraries.
MattGPT says:  that will use in python, at least that more of them. But there's 3 that I use before, and I think I would recommend you to try Camelot and Pabula and PS. One
MattGPT says: tabular is actually the oldest one. It's a Java-based library, and then it's got a type and wrapper on top so all that could be good, right, because Java stuff tends to be somewhat robust, but it can be a pain to use. But like
MattGPT says:  So here's I just downloaded that particular thing. All of them will actually go out to a yeah URL as well. So you can read about what tabular does. But basically it's really simple. You just import it.
MattGPT says: and then you can just say, Read Pdf. And you give it to pop the Pdf
MattGPT says: and if it's a 70 page document, don't just say, read the whole thing. If there's a table skip on specific pages that you want.
MattGPT says: you just give it a string which is a common separated value of the pages.
MattGPT says: So we know looking here in this Pdf, here.
MattGPT says: But on page 4,
MattGPT says: there is the table of this table right here.
MattGPT says: Okay, and that could be useful for the years from 2010 to 2021
MattGPT says: with fatalities, a bit like the data that you have, and then
MattGPT says: things go down on page. That's end
MattGPT says: 7. Okay? So we can look at page 7, 2. But on page 13 there was one in particular I thought was interesting.
MattGPT says:  yeah. So here is a comparison between 2 years. Okay, so possession and fatality rate at the state level
MattGPT says: path of the fee 3 is
MattGPT says: okay. So
MattGPT says: if you run this here
MattGPT says:  it will read the Pdf. And it will look at those pages there and then it tries to extract tables there.
MattGPT says: Okay? So you can see that we get a list of 2 tables and pabula automatically turns them into a data frame. But you see that it doesn't do very well on that first one.
MattGPT says:  but it does. Okay on the second one.
MattGPT says: Okay. So sometimes, depending on your table, you might want to chop across these different libraries.
MattGPT says:  but you can see here, but we've at least got the data into now a form that we can look at and say, since we've done a pretty good job.
MattGPT says: but we don't really want the national summary data at the bottom. And so we can drop those rows.
MattGPT says: Okay? So this could be an instance where you actually do use robe indexing and just drop 50. The Camelot. A similar sort of thing. You can read the documentation for that. Let's try and see what we get. If we run the same.
MattGPT says: Pdf, look for those pages. Page 4, on page 13.
MattGPT says: It also by default is not finding any tables on those pages.
MattGPT says: Okay, but I have an algorithm that's looking for certain things in a Pdf.
MattGPT says: so that's not great. But if you read the documentation, you can see that it works in 2 different ways. It works in like a stream based thing which is assuming that there's regular spacing between columns, and it's not looking for lines or anything like that.
MattGPT says: And it's the bold version, though, is that there's going to be grid lines around the table called the lattice flavor
MattGPT says: and see, it doesn't find, because these, if we go back and look at these particular tables.
MattGPT says: okay, they don't really have black lines or any sort of grid lines around them. Okay? So it doesn't find anything there.
MattGPT says: Yeah. So if we have black lines like it won't work it, it will. It will work by default, right? But then, if if you have stuff where you don't have any black lines, you can try the stream option. Okay? So now we're gonna try the stream version.
MattGPT says: and then you can see.
MattGPT says: Oh, it's not who tables now.
MattGPT says: Okay? So the the fault is assuming that any media is, gonna have, like
MattGPT says: a lattice, represent
MattGPT says: actually looking for lines in the media, and then saying within each of these cells. Let me try and pull out data right? Whereas the stream flavor is saying, no, that's just spaces, and this is, there are kind of lines. But then it's not gonna capture those. So it's just seeing all this consistent spacing there.
MattGPT says: And so this looks like a table. And as soon as it finds something that looks like, hey? Well, it will try and extract for us. And so this gives you back a list of objects.
MattGPT says: And then the dot, yeah, yeah, actually, there will be data from. And you can also then save it to Csv and stuff like that.
MattGPT says: So it's not an okay job. Right? It's had trouble with the headings, and it puts it across multiple rows. But when it actually gets down to the regular data here, then we're getting columns there, so we'd have to do some selection here and tidying up of this. But we've at least managed to extract the data out of the Pdf.
MattGPT says: In a format, and then you can see at the bottom. Yeah, the table itself has this change up in there?
MattGPT says: And then the second table again. It did a reasonable job for some of it. But for some of this stuff like it split this across 2 rows.
MattGPT says:  so you can still work with this data. It's better than having try and type it all in again, right? Or do and copy a paste
MattGPT says: that Pdf plot. Oh, I missed the URL up there. You can just do all that. But this works slightly differently. Here. What you do is you?
MattGPT says: Open the Pdf. And then you get an object which is the reader for that? Okay? And then you do. You look, you identify, you select specific page. So the Pdf, you say pages, and then
MattGPT says: so 0 for the full page would be 0 one for free to get the page. And then you can try the extract table function on that specific page
MattGPT says: saying, focus in on this page, extract the page
MattGPT says: and then pitch 14 really like it.
MattGPT says: so then, what we get out of here is that we don't get a data frame. We get a good old
MattGPT says: list of lists right? So each row, each the go list, and then each row is a list within that. And each item within the list is another column.
MattGPT says: Okay? But it's done. And really nice job. There, I think of pulling out the data.
MattGPT says: and you can work with that table 2 just on page 13.
MattGPT says: But a list of lists. And the first item in that list is a list of 3 things which are the current headings. So it's really easy to turn a list of lists into a data frame.
MattGPT says: Just say Pd data frame. And then we don't want the first item in the list, because that's the column heading. So
MattGPT says: have all the data from row 7 row fruit or the second diamond. That's proof the end. And then set the column.
MattGPT says: That is to be the first one.
MattGPT says: And then, hey, we've got our data.
MattGPT says: Okay. So there's there's freeways or freely, and tools that kind of get to the same thing.
MattGPT says: I can't say I can't say, always use tabula or always use Camelot, because it really depends on the media right? But they're all basically working the same way. So I'd say, like, use this as a template
MattGPT says: when you come across stuff. you can see that. Okay?
MattGPT says: So that's
MattGPT says: that's extracting data from Pdfs.
MattGPT says: this was another little thing that I come in full of the other day on the Home, and I think I'm showing you this book in notebooks, but like
MattGPT says:  just in in notebooks when you're working with them on Jupyter Hub. This doesn't translate particularly well, it's saved. Gonna go to the Pdf. There are ways of getting this, but this is sort of like a time step towards interactivity in a dashboard kind of way.
MattGPT says: Okay, so there's a really nice set of things that widgets
MattGPT says:  and so
MattGPT says: basically, how this works.
MattGPT says: you know, it's very simplest is that if you have a function that does something and produces some output like here, so that this function here defines power. So we want to get in power. You know, 2 to the N basically. And when our function is that we give it n, and then we'll just print out 2, for the power of N is whatever
MattGPT says: through the package. Okay, so you can see what power does right in this function here.
MattGPT says:  I actually love this this out and this type of stuff.
MattGPT says: Okay, if I was to run is the fun.
MattGPT says: find this function?
MattGPT says: Okay? And so now we've got a function. And then we could use it by saying, like.
MattGPT says: what is 2 squared? Right? So 2 to the power. 2 is 4. Okay, so that's just, you know, it's about basic using. So the nice thing is that
MattGPT says: this
MattGPT says: you can use the interact widget. And from Ipi widget, this is called a decorator, which is kind of more advanced. So I mean, stop. But basically, when you have an app, yeah, but you put it above function. Okay, to say, make it interactive.
MattGPT says: And then here, in the argument that goes with this, you specify the values that you want to be displayed as interactive. Okay? So we're saying, but we're going to have N. And N. Can go between one and 10.
MattGPT says: And so then when you run that
MattGPT says: you can see that it sticks in a slider widget right? And you can. You can specify where to start yet as well, actually in there. But like here, we're just it's going to start
MattGPT says:  And then
MattGPT says: you can see that every time you move the widget. it's observed, and it calls the function. So you can see? Yeah, strange question. But like, is there any way to make that like
MattGPT says: like, I don't know what's the word
MattGPT says: I guess. Like a spectrum instead like that. That like goes through increments from, yeah, yeah, if you if you look up.
MattGPT says: yeah, I didn't put the link into there. But yes, there are ways to do that. If you look at the interact, function from I 5 widgets.
MattGPT says: 1.1
MattGPT says: are. yeah. Oh, wow, okay, yeah, sick. And so it's just yeah, that's sort of the influence.
MattGPT says: So this is like a range of things. So you know, it's any function, right? So and let's just think about like what we were just doing.
MattGPT says: Assignment 5, we had the world map. We had this life expectancy data frame.
MattGPT says: Okay, so let's say that we want a function that will give in a year. Select just those rows for all of the countries. Most same with our map, our Geojson file.
MattGPT says: We have to do this? Question. 15. Find the right columns, and then just do the plot of a life expectancy. Connor, okay. And
MattGPT says: the thing that's missing from this, obviously is that the maximum and minimum is going to change year by year. But you can see that
MattGPT says: it's then going to give you some sort of interactivity on this.
MattGPT says: So it's it's kind of funky in some ways.
MattGPT says: But it's really for data exploration. If you want to play around with your data
MattGPT says: use this indirectly. So any nice kind of next step up.
MattGPT says: Okay, how we do. Oh, we got put in that. Okay.
MattGPT says: we're gonna keep. I think a number of projects from the sound of it are gonna need to some level of web. Okay.  And so I want to talk about that. Yeah, birthday a bit more, and then maybe next week.
MattGPT says: but depending on what it is that we want to scrape, there are different approaches.
MattGPT says: Okay? So here's an example.
MattGPT says: Something like, yeah, the billboard 100 hot 100 end of the year we get like
MattGPT says: the talk singles.
MattGPT says: Wikipedia is one place you can get it from billboard.com. But Wikipedia doesn't. Really. You can find data on Wikipedia.
MattGPT says: That's great, because it's 1990 S style web design right? And they never came to that. But I think they never will change that. That's the style
MattGPT says: so where there's data, it tends to be in these sort of, and they greet them, and they used to be even simpler and just structured. They now use sort of ones where
MattGPT says: a lot of the time. They're like the Wiki sort of all ones where you can actually sort of click on this and get them sorted in different ways.  but that's still really kind of simple structure there. So
MattGPT says: the first place to start actually is equivalent to what we were just looking at with
MattGPT says: before you have to jump into the HTML, let's just do that.
MattGPT says: Here's our page. Yeah. URL,
MattGPT says:  is, if you've got panas loaded.
MattGPT says: Okay? And you have a web page that has some tables on it.
MattGPT says: Then you just say, Pandas, read HTML
MattGPT says: and give it the yeah, URL.
MattGPT says: It will do the equivalent of what we saw with tabular, and
MattGPT says: that
MattGPT says: oh.
MattGPT says: expecting a specific format.
MattGPT says: Good
MattGPT says: of year.
MattGPT says: Okay? So now, what do we have? We're just using Panda's read, HTML,
MattGPT says: then tables. Okay? So we've got 2 tables on this page. If we go back to that.
MattGPT says: think they're like, there's one main table that I'm seeing there. But like.
MattGPT says: Oh, there's a table down the bottom here, which is navigation.
MattGPT says: Okay? So if you take a look at a but then they won't. First one
MattGPT says: you can see. Bam. Okay? So it's got us. It extracted our billboard. 100 child.
MattGPT says: 3 main columns. We've got our data in there. Okay, so it's really simple, really straightforward. See? Especially if you're
MattGPT says: data. Looks like it comes out of a database or something like Wikipedia, where it's really, consistently.
MattGPT says: You don't even have to get into scraping and just try to read agm out from them.
MattGPT says: Give it the yeah, URL, and you'll see what kind of people should get out of it.
MattGPT says: Okay, so that's a kind of simple.
MattGPT says: straightforward sort of way.
MattGPT says:  but remind me how many people have done any HTML stuff before it's so about okay.
MattGPT says:  so that. But there's some data in this
MattGPT says: in this table, which we're not getting from that. Okay? So you can see in this case. Each title is linked because it's another wiki and entry right? So that he waves, is linked to it has its own page
MattGPT says: right? And there may be some data in here that you might want to get. You were doing something where you were starting from the bill for 100 per specific year. And then you wanted to for the top 50, the top 20 songs. You wanted to get information about how it charted
MattGPT says: around the world. Right then the Wikipedia Page is going to be a nice one, that you can jump to that page and then extract the table here, and sort of have highest rank white countries, or something like that. So you can see how you could get a secondary linked data set from that.
MattGPT says: And then all. You might also want to collect some data about the artist or the artists. Right? And then again, there's gonna be some data. Here's in a table form or stuff that you might want to pull out.
MattGPT says: So we don't get those
MattGPT says: those yeah urls here. Okay. any web page where you've got. You're trying to understand how it's structured
MattGPT says: in any of the browsers. Now, if you right click and to inspect.
MattGPT says: and then you'll get a lot of stuff from a lot of Javascript and Css. Information. But over here you'll get a browsable view of the underlying HTML.
MattGPT says: Okay? So the structure of the page
MattGPT says: and Wikipedia. This is why Wikipedia is a nice one. It's great, because it's really straightforward, simple HTML that they use.
MattGPT says: But I'll show you some other examples which are a little bit more complicated. But if you go back to sort of to notebook, you can see
MattGPT says:  the example here, right? So those
MattGPT says: those rows. Here you look at the underlying HTML,
MattGPT says: you can see there's a lot of stuff at the beginning, header, but you'll find the point, and if you right click on a specific point should take you to that. The point here, when you're observing it
MattGPT says: somewhere, there is a table, Helen. Okay here. So less than is beginning of element and table. Then it's class and wiki tables sort of all
MattGPT says: and then, under defect.
MattGPT says: it's got some items. Okay? So the way. I think we came out of work that we haven't designed for
MattGPT says: open tag. and then the backslash, and then same element, name, and then nested within that you could have
MattGPT says: hit 10
MattGPT says: understanding sort of that structure. Then there is a
MattGPT says: in this case. Actually, we've got this Teabody, the able body, and then Trs, then a rogue.
MattGPT says: and so then rows. And then you've got 3 header columns that have the title number title of Artists.
MattGPT says: Okay? And that's then corresponding to that first row right there, right? But I'm honest. So that's in the table heads and
MattGPT says: top row. And then the second row is where the data starts. And so then each row, each er element
MattGPT says: highlighted
MattGPT says: yellow, that has free TV elements in them, too.
MattGPT says: where the data is.
MattGPT says: And so once you understand that structure. There's a way to sort of navigate around
MattGPT says: and get at that. And so
MattGPT says:  yeah, but plans probably to go through all of this. But the basic steps are
MattGPT says: your you've got your base. You've you've got your yeah. URL, okay? And so the request module and type is one way to basically do what a browser does, which is go off to a URL and bring back the underlying Http data to that.
MattGPT says: Okay, so if we run that
MattGPT says: you've probably only come across responses when you've a page is missing, you get something like 4 or 4 or 400 based on time. A response. 200 means, there was data there. And I got back some data.
MattGPT says: Okay, and what you get back. If there's actual data, then it's the H, the underlying HTML, okay? So we can get back a string object which we look at like the first 1,000 characters of that
MattGPT says: you'll see is
MattGPT says: in stream form the HTML that we saw when we did the expect behind that, hey? We could work with it like this, but you know it's a long string, and it's really HTML. So what we wanna do is that we want to turn it into a python object, which is, that, we can navigate same sort of structure that we saw there
MattGPT says: when we did inspect right here.
MattGPT says: Okay? And then we're able to see. Oh, yeah, we're inside this element right here. And we can find this Pr.
MattGPT says: and there's the first TV is there, and second, TV is there. But inside that, then
MattGPT says: there is a a hyperlink.
MattGPT says: And that hyperlink has some information that we're interested in, which is, we actually don't want just the content here, which is what we got from the movie. HTML, but we want to be able to pull out the the link there as well.
MattGPT says: Okay. so
MattGPT says:  we'll come back to this.
MattGPT says: But basically you can turn an HTML string into a python. Object that you can then navigate, and that's the drop of it.
MattGPT says: So then, understanding the structure of that, we can just say, find the table element within that HTML object
MattGPT says: which has a class attribute Wikitabal sort of okay. So what it's saying is.
MattGPT says: talk about that. Yeah, here's the table element. It has a class attribute which is with the table space sort of
MattGPT says: okay, so that will know that will navigate us to this part right here.
MattGPT says: and then we. It's a object that we can walk around. We can then find all of the T. Rs, so come underneath that kind of in that big, hey? Those are all the rows that we want.
MattGPT says: So if we run that, then how many rows do we have? We have 101 rows. We're not interested in the first row.
MattGPT says: because that's our column headings, or we need to treat the first row differently. But then we have 100 rows of data, which is the 100 items on the top, 100.
MattGPT says: Aye. So then, you can write some code that looks like this from python basics
MattGPT says: move through each of the rows from Row 2
MattGPT says:  and find. That's the elements that we want within that which is the TV elements that's free columns, each cell at the time.
MattGPT says: And then we can pull out data it. So you can see that we're creating a dictionary where we want data from the first column, which is Peter, we ranked second title for either artists, but we also want to add into our piece of information, which is the Wikipedia links to the song URL and the artist. Yeah, URL,
MattGPT says: thanks. So we could do that. And then you can see that we end up with a list of dictionaries
MattGPT says: where we've got the. We've scraped the data on and pulled it out. Put it into the pipe condition.
MattGPT says: Thanks. But this dictionaries is really nice, because if you give that for pandas.
MattGPT says: Pd, dot data frame.
MattGPT says:  and we'll get a data out of that.
MattGPT says: I don't know other phones and move that agent.
MattGPT says: Okay? So, depending on what data you need to get out of there, you can get away often just by saying, Okay, let's see what kind of if you realize like that it's not doing a great job or the table instruction in such a way that I need to
MattGPT says: kind of break it down. Then I would jump into sort of thinking about doing this basic kind of scraping where you basically just load the whole page and then turn it into a python object.
MattGPT says: Beautiful super object is the name of the module for that, and then you can extract them to make that
MattGPT says: okay, so we'll come back to doing this on on Thursday, and then we'll talk about the next level up, which is what you may come across a lot more these days, which is like single-page applications. Those are websites where you go to them.
MattGPT says: And if you loaded them with this approach you'd get no data, because what happens if page skeleton loads and then everything out, the page comes through the Javascript period.
MattGPT says: Okay, so there's a way to get there's a number of ways to get the data out that way. One of the browser automation, which is where you drive a browser from Python
MattGPT says:  and you can then have it go to the URL. And then you figure out, okay. If I was to get this data myself by clicking, what would I do? Type in 1,990 enter.
MattGPT says: and then we'll have another fill out this table. And there would be like 10 rows of data running. Copy those rows of data and then click next next, next, next, next next next, until they were running all of those sorts of things some of you that are thinking about like going to a web database and looking for specific reports, specific restaurants, specific locations, or state. For some of that, you can automate all of that as well are using this.
MattGPT says: Okay? Sorry. But then go over that.
MattGPT says: Those are the things that I'm really excited about how the
MattGPT says: communicate amongst themselves, but also as a way to communicate with me. Okay, so you can just ask questions there. And I'll try and put look at those and give you answers to that. So do questions at this stage. It's sort of like, is the focus of this project good or would think
MattGPT says: we can use these data, but we're not sure how to access them. If you can give me a yeah URL, or some idea of that, then I'll look at that. So what I want to kind of start to shape some of the material that we covered going forward on the questions that you're working on, and the kinds of things that you're going to need to do.
MattGPT says:  the one actually I mentioned last week. But next a week on Thursday we'll do the initial proof presentations. Okay, so we'll use the whole session a week on Thursday
MattGPT says: to go through. And
MattGPT says: you know each group can give a 5 min presentation. This is what we're working on. Here's the data. This is what this is, what our fan analysis is going to be.
MattGPT says: and then you'll give feedback very
MattGPT says: next. But next Tuesday
MattGPT says: I love Halloween, so that'll be in part, and there'll be a Halloween relays activity.
MattGPT says: I will try and come up with some sort of data related costume where you want to costume. And do, okay, yeah, massive presentations. Thursday or Tuesday. No.
MattGPT says: next week. Next week. Next week. Yeah. Okay, got it. So don't this time proposal got it? Okay. yeah. Do we have assignment? Do anything?
MattGPT says: Okay. So for you folks on zoom sorry for the delaying getting started, that was.
MattGPT says: And as you're getting
MattGPT says: the folder, but today posted but in Jupiter hub you'll see
MattGPT says: folder called 10 B.
MattGPT says: examining data over time.
MattGPT says: Okay, there is a 10 A
MattGPT says: that I will try and get posted in a minute.
MattGPT says: Some reason that everything that was 10 that was too much
MattGPT says: data, and the the size of the files were too large. And so Jupiter Hub is
MattGPT says: not letting me post it.
MattGPT says: So I split, tried to split it into 2, and I'll post 10 A. And 10 B. We're going to get started looking at folder 10, a
MattGPT says: okay, And you can see in here.
MattGPT says:  that there are 2 notebooks that relate to this one data set about rat sightings. Okay, so
MattGPT says: in New York City you can take a look and at the link there and find out about it. This is
MattGPT says: calls to free 1 one in the New York City area
MattGPT says: over time.
MattGPT says: people citing, reporting that there's a rat in their building on the block that they're seeing on the street.
MattGPT says: Okay. So the first thing you'll want to do is wh what your task is.
MattGPT says: I'm here in
MattGPT says: in the room. People are either working on their own or or in pad.
MattGPT says: So if you can go through and
MattGPT says: take a look, understand the context of the data. If you remember what we talked about last week
MattGPT says: on Tuesday, about asking questions of data from
MattGPT says: the chapter? where do these data come from? Who manages them?
MattGPT says: Why are they important? What are those sorts of questions? So you be thinking of those? But what I want you to do is use this template that we've been playing around with  to doing the initial exploration. Okay? So using pandas.
MattGPT says: you'll load in the data. Okay. wrap data frame.
MattGPT says: It's in the data folder. We'll wrap sightings with today's date on it.
MattGPT says: Okay? And then, so then, once that's loaded.
MattGPT says: we know the things that we just go through the
MattGPT says: normal steps that we've got there, how many rows, how many columns?
MattGPT says: Take a look at it
MattGPT says: so you can get a sense of what's there in the data? It's got 38 columns. So you won't see necessarily all of those.
MattGPT says: So you want to get a sense of what those those are.
MattGPT says: and then
MattGPT says: understand those. Okay? So if you can. If you can get working on that, if you've got questions, you just have to stick them in the chat, and I'll try and look out for those.
MattGPT says: The second notebook here is giving you
MattGPT says: some examples of how to do each of those steps. Okay, but don't look at this one first. I'd like you to try and see if you can fill in the steps yourself. Alright. This is just practice stuff, and this will be practice for assignment 5
MattGPT says: and the other things. But before we came online that I mentioned. there's no assignment for this week
MattGPT says: but there is a quiz which is due on Sunday, so that should be posted now
MattGPT says: on this week's readings, Simon 5 will be posted soon, and that will be due a week on Sunday. And what we're doing here, and what we've been doing this week is sort of practice for those steps.
MattGPT says: Okay.
MattGPT says: okay, so on. Zoom. If you want to work together, just put a breakout room with you, the 3 of you in there, and you can join that breakout room, and then you should be able to
MattGPT says:  talk to each other and work together if you want to.
MattGPT says: I mean, I could stop. But you have any questions on how to move through each stage. Okay, yeah.
MattGPT says: started off. So I'm gonna check open.
MattGPT says: Yeah, that that's that's awesome. Yeah. Start
MattGPT says: I would just sort of like. close the tavern.
MattGPT says: It's like.
MattGPT says: okay, so try and close down that.
MattGPT says: Okay, yeah.
MattGPT says: So don't do that. Yeah. Just loaded in as a data frame
MattGPT says: last week? But when I get
MattGPT says: a big thing.
MattGPT says: yeah, did that come back again? Yeah. So then I just like, II hook something nice and yeah
MattGPT says: accountable regions.
MattGPT says: Yeah, here's again. Sorry. Just validate it once in 2 min. I'm not much arguing.
MattGPT says: Good.
MattGPT says: I think it would be well, that would be
MattGPT says: you try clearing your booties.
MattGPT says: But this site can do that, too. Not a settings. Privacy, opening
MattGPT says: seeds, hearing?
MattGPT says: Very, very good.
MattGPT says: Yeah. And then,
MattGPT says: yeah, that's the one. Then you want to find okay?
MattGPT says: and maybe maybe you are just who you pen alright. So go into click on the U. Penn. Oh, yes, I click under the
MattGPT says: can you delete everything which is under this domain? Yeah.
MattGPT says: Okay, I'll try.
MattGPT says: Yeah.
MattGPT says: Okay.
MattGPT says: so that's that's sometimes.
MattGPT says: But I wonder if it's just something. Yeah, try. And again.
MattGPT says: that helps event.
MattGPT says: Okay, yeah, no, it's not your fault. I don't know why even that is, it will be something about
MattGPT says: just make sure.
MattGPT says: Correct.
MattGPT says: I'll send.
MattGPT says: Okay.
MattGPT says: yeah. Okay. So when you do value counts, can you try to do that? Said to that problem. And
MattGPT says: what would say.
MattGPT says: and
MattGPT says: percentage.
MattGPT says: That's so.
MattGPT says: yeah, I mean, you want to get a sense of data right? So there's different ways to do that. So which column are you looking at? I will submit that
MattGPT says: that's the company
MattGPT says: which. Well, okay, well, what you can do is I just take like.
MattGPT says: let's see, the one that we know has some Nas in it like closed date. Right? I guess, is one that seems to at
MattGPT says: matters. Thanks.
MattGPT says: So you can see, like, you know, just by looking at it. Or you're taking a random sample that there's a mixture in that right? So there's 2, there's different ways to do it. So remember, we you know, you could use value counts. But this is not great data for that. This is timestamp data, right? So that's going to give you like loads, you know.
MattGPT says: loads and loads of different values in there. So for this
MattGPT says:  one thing that you can. There's a couple of things that you can do you can do
MattGPT says: is an a okay, which is like a filter.
MattGPT says: And that will be true.
MattGPT says: if the value is missing. right and false, if it's not so, you can see that this we've got.
MattGPT says: We've got that. But all 230,000 rows. So then, true and false. if you sum them up
MattGPT says: true, will be
MattGPT says:  true is one, and false is 0 right? So
MattGPT says: that's a good way of with this kind of data, saying how many rows have missing values
MattGPT says: in that column?
MattGPT says: Okay, so there are 17,600, a 2
MattGPT says: rows where there's no closed date.
MattGPT says:  And so then, you know, the opposite of that is how many does that? So you know, that's that's one way to do it like looking for
MattGPT says: missing values. another way that you can do it is.
MattGPT says: let me think these counts.
MattGPT says: Yeah, so counts
MattGPT says: is another way of doing it which gives you the opposite. This is account function on a specific column
MattGPT says: tells you how many.
MattGPT says: how many rows have a value in that column. So NA is like, not, you know, is missing value. Basically. So we know, like in the shape of the data frame, right? That we've got
MattGPT says: for that. Yeah, for that column. Right? Yes, yeah, basically that right? So you can see that. That's how many rows there are.
MattGPT says: That's how many missing for closed date.
MattGPT says: or the inverse of that is by count.
MattGPT says: Okay? So usually, II you know you can. And you see, some people do this. But when you've got a data frame with like loads of columns like this.
MattGPT says: Then.
MattGPT says: you know, you see, some people straight away do something like that, which is okay? Right?
MattGPT says: Yeah, I mean, maybe this is good way. Actually, maybe I have a good thing to
MattGPT says: a way to think about the columns. If you just apply count to the whole data frame, you can either apply it to an individual column or a selection of columns, or if you just click, count on the whole data frame.
MattGPT says: And you know how many rows you're expecting. Right? This is this is showing you for the unique key.
MattGPT says: variable column. How many have
MattGPT says: values in and so on.
MattGPT says: Okay? And so you can see
MattGPT says: which ones where there's there's missing missing values. And then you can figure out by how many right.
MattGPT says: And you could take this.
MattGPT says: Yes, you can even do
MattGPT says: see if this will work out
MattGPT says: right? So that's the number of rows. Take that away from the numbers
MattGPT says: that have values in them. So you need unique. He has no missing values, created 8 has no missing values.
MattGPT says: Fluffy have that many location type.
MattGPT says: Then 12 rows out of that 23. That's 120,000 that have missing values.
MattGPT says: Zip code is a pretty good one, too. Right? So it's just a way of sort of seeing like.
MattGPT says: okay, which of these variables? Whereas the intersection street one and Intersection Street 2,
MattGPT says: almost half of the roads don't have something there.
MattGPT says: but Cross Street seems to have that. Pretty well.
MattGPT says: Landmark
MattGPT says: facility type. Those are not useful variables in the sense that they haven't got a lot in them.
MattGPT says: there's some things in this data in this, in this data sheet
MattGPT says: that doesn't seem relevant at all to the rap. The rat sidings right? But they're still in the data sheet because it's like vehicle type Taxi Company borough.
MattGPT says: where you know some some locations which are to do with transportation. And this, remember, the context of this is the free 1, one
MattGPT says: database of calls that come into 3, 1 one. So you know, you got something to report to the city of New York.
MattGPT says: Equal 3. 1, one could be a wrap signing could be.
MattGPT says: you know, a pothole. It could be a
MattGPT says: that you wanted to report a rogue taxi driver, driver, or you know any of those things like anything that you you can go, and those all get recorded in a way. And so what this data sheet is that they put up on
MattGPT says: open data. New York is actually an export
MattGPT says: for just the calls about rat sightings or rodent sightings. Right?
MattGPT says: And so what you often will find is that you get these columns which actually had nothing to do with.
MattGPT says: They're they're always blank for that particular type. But they're in the database.
MattGPT says: Yeah. So that would be the thing to do. What you would do here would then, you know, identify the columns that are interesting or useful.
MattGPT says: You could get rid of the ones that
MattGPT says:  for example, you know, normal crowns
MattGPT says: 75 is 0,
MattGPT says: and we just gave
MattGPT says: you the the mugs. And
MattGPT says: here.
MattGPT says: like facility type.
MattGPT says: I was supposed to find that.
MattGPT says: Yeah. So 0 means that there are no.
MattGPT says: If you do count on that right, that means that there are no rows of values in, so all of them are. Na.
MattGPT says: so I thought the account included the na as well.
MattGPT says: No, so account is the inverse right? So these 2 that should be here. So is na is a test to say, is the value missing right? Whereas Count is the inverse of that. Tell me how many rows
MattGPT says: have values
MattGPT says: 0 value.
MattGPT says: And that returns.
MattGPT says: Thank you.
MattGPT says: yeah, so
MattGPT says: and
MattGPT says: right. So account that, you know, that's the filter, each of that so? And a quick way to then find out how many is to sum those up. That's true.
MattGPT says: all of the that adds up to the number of rows. So facility type is always know.
MattGPT says: always missing.
MattGPT says: This count gives you the inverse, which is, tell me how many rows. There isn't a value in this
MattGPT says: 0, right?
MattGPT says: So in some ways, probably this is easier to to use right if you think about it and doing what we did here
MattGPT says: actually isn't a bad way of getting started right. And remember, you could do something like
MattGPT says: rat ef dot shape, and that gives us back a pair, a list, and the first one is the number of rows. Right? So
MattGPT says: that's just a way of saying, you know, get me the shape of the data frame.
MattGPT says: Tell me how many rows is in it that's in that 0,
MattGPT says: and then take away from that.
MattGPT says: What's the difference to the result of counting up how many roads there are. So anywhere where there's 0 is then
MattGPT says: perfect data for that point.
MattGPT says: and anywhere where it's like.
MattGPT says: you know, close. Like. Given the size of this data set, the X coordinate is going to be useful right?
MattGPT says: And and then these ones, which are really low missing values. This is showing that you know.
MattGPT says: community, the board is good. Borough is good. There's only 4 rows out of 230,000 where it wasn't recorded wherever it was, Manhattan, Bronx queens, and so on. Brooklyn.
MattGPT says: But, like you said, these ones are totally useless because they're completely empty. Right?
MattGPT says:  So you could do something which was be like, Okay, I've got a threshold. Let's say, let's say that we pulled this like
MattGPT says: missing counts or something.
MattGPT says: and
MattGPT says: so we've got that right? And so you could say
MattGPT says: anything which is more than a thousand missing values.
MattGPT says: Okay? So those would be like your subset of rows, a subset of columns
MattGPT says: where you've got less than a thousand missing values.
MattGPT says: And maybe that's too, you know. Maybe given the number of rows, you may want to say 10,000,
MattGPT says: so that you keep like some of these in.
MattGPT says: Yeah, maybe 10,000 is a good sort of thing that
MattGPT says: okay? Now, remember, whenever we do like count or any of these functions, what we get back is we get back a table where we've got an index.
MattGPT says: It's labeled here, and we got a value. And so it's been when we see it's kind of like a dictionary.
MattGPT says: but pandas is called an index there. so you could then say
MattGPT says: an index. And there, you see, that's the list of columns that have
MattGPT says: less than 10,000 missing rows
MattGPT says: in that column. Right? Then I might do something like
MattGPT says: How's to beep?
MattGPT says: Look at that as a list.
MattGPT says: And then, if you apply that to your
MattGPT says: okay, you get up. Let's pull it.
MattGPT says: This is gonna be the first.
MattGPT says: Sorry. Say that again.
MattGPT says: Yeah, alright. So let's look at this again, whenever, like all of these things that we
MattGPT says: oh.
MattGPT says: and whenever we do anything like
MattGPT says: location.
MattGPT says: the
MattGPT says: Albarn, yeah.
MattGPT says: Okay. So borrow is a categorical, variable right?
MattGPT says: And so it's a good one to do value counts on.
MattGPT says: Okay, sand.
MattGPT says: We get a tabulation like the most frequent boroughs. Brooklyn, followed by Manhattan, followed by this kind of structure. Where you see this come out. This is, you know, like a
MattGPT says: this is a distribution sort of table or account table. The left-hand side is called the index.
MattGPT says: Okay, and the right hand side is called the phones. Right? So
MattGPT says: what we're just doing here is. If you say, then index. then that just gives you those labels
MattGPT says: there. Okay, so that's what we did right here. The steps of what we did here was like missing counts. Was this,
MattGPT says: this data of this object that we have created.
MattGPT says: which
MattGPT says: again, it's a sort of table, right? This is all of the columns in the data frame and telling you how many rows that are for each column. for each variable where there's missing values.
MattGPT says: Okay? So if we wanted to just get the list of
MattGPT says: of those right, and we could do that. Well, that's just the list of all of the columns.
MattGPT says: We did that? 5
MattGPT says: subtracting the yep, that's right. Yeah.
MattGPT says: So you could kind of do the inverse, I guess, which would be like.
MattGPT says: You know, you could just do count on the column, on on the whole data frame. Right? Right? Data frame
MattGPT says: count, which tells you how many?
MattGPT says:  and so you can say, like I. You could. Then you could do the the math yourself, which is like, I only want columns where it's greater than
MattGPT says: 220,000
MattGPT says: like that. Okay, so every time you do a test like this
MattGPT says: that gives you
MattGPT says: look through, it gives you a filter
MattGPT says: for that. Okay, so
MattGPT says: then we need to.
MattGPT says: You would need to actually
MattGPT says: do that again to get those ones there. Do the selection.
MattGPT says: Okay? So those are just your columns where?
MattGPT says: So you could you could do an either way, you know, calculate the ones, the number missing, or calculate the number
MattGPT says: that are which have values in them, and then
MattGPT says: specify it, anyway. But that's what we did there, right
MattGPT says: right here. So we
MattGPT says: create this table count, and then the filtering it by the value must be for the missing one, the value must be less than 10,000 value.
MattGPT says: and to get the list of columns that we then keep, we say, give me the index.
MattGPT says: And that gives us a list here. Okay, so we've got
MattGPT says: these are columns out of our data frame where there's enough data here like 200, at least 2,000 220,000 values in that.
MattGPT says: And then what we could do is that we can say, Okay, I don't want to bother every time I'm working with the data frame. I don't want those columns I'm not interested in. So do a subset of the columns. Okay? So you remember how you subsect columns is the unit
MattGPT says: user, create a list of the columns that you want to keep
MattGPT says: just do a selection like that. Okay? So now then, rep 2. Df.
MattGPT says: it's gonna have less than 36.
MattGPT says: Now, we feel it's just 21 columns. Right?
MattGPT says: Okay? So yeah, that's a good.
MattGPT says: That's a good sort of approach. When you're suddenly, you're dealing with data, and it's sort of
MattGPT says: it's a really common thing. I think, that you'll find with a lot of this open data is that
MattGPT says:  they're often
MattGPT says: that they're often come from a much larger database, and there's
MattGPT says: stuff that's been that's in the database, but isn't relevant
MattGPT says: or fully, you know. Don't have all the roads in there.
MattGPT says: All of this stuff like taxi company borough, and all of that must be for a completely different sort of Freemont report.
MattGPT says: The other thing that you often find is that
MattGPT says: you have data that's been added to the base data that you're interested in, because for some applications, you know. And on open data, Philly, we saw that sometimes you'll see there's a Csv file. There's a go, Json file, but then it will say, sort of like HTML, or you click on it, and it will be a map that you can actually investigate yourself. So they've kind of posted a dashboard of some kind.
MattGPT says: and often to make dashboards work. Then you have to add in extra columns can do certain things right. But they're not really relevant for data analysis. So getting a sense. Yeah, this is already just part of
MattGPT says: working with real data. And it's often gonna have a mixture of the stuff that's interesting and useful for your question.
MattGPT says:  right? So then, let's say we're complaint type.
MattGPT says: We come back to this and look at complaint type
MattGPT says: and
MattGPT says: an hour.
MattGPT says: Okay, so that's right.
MattGPT says: When it's categorical. Remember another way that we can get at it is like, what are the
MattGPT says:  spell it right? Unique.
MattGPT says: The unique function will tell us, like.
MattGPT says: what's the main values? Okay? And so in this, in this way.
MattGPT says: There's only one value in the complaint type. So it's useful. This is useless column, right? But this is polish. This kind of confirms the fact that in the free one wine database they'll be like, what's this complaint about? It's about rodents, it's about taxis, it's about potholes. It's about
MattGPT says:  but mail. Whatever else can be complained about, or can be reported to free 1, one will be in there as a pipe.
MattGPT says: but when they exported this data, put it up on the new open data site. Then they just subsided the database and said, Give me all the road and rows.
MattGPT says:  And so that's why we're getting these things, that
MattGPT says: and
MattGPT says: the vehicle type, the taxi pickup. Those are for a completely different type of report.
MattGPT says: Okay? So
MattGPT says: yeah, it's a good sort of step to get to that. So then, when you get to this question. After that
MattGPT says: we've narrowed this down.
MattGPT says: you know. We would probably then say, well, we would drop complaint type right out of this as well. But this is our subset of ones that we think are useful. Then we'd look at these and then say, Okay, so what
MattGPT says: agency might be another one that you might want to look at, to see.
MattGPT says: Is that another one where
MattGPT says:  there could be all different sorts of agencies, that
MattGPT says: yeah.
MattGPT says: and all.
MattGPT says: I don't know what? The OHM. That's cool.
MattGPT says: A
MattGPT says: commonal health and
MattGPT says: mental health, maybe. Oh, it's an agency
MattGPT says: and mental hygiene. Okay. so agencies and agency name, I guess those again again be like varied in the the larger data set. But in this case
MattGPT says: they're they're not. There's no variation in them. And so
MattGPT says: that would be then another step to say, well, complaint type, there's no value. There's you know, I can't use that agency. I'm not going to use that agency name. I can't use that right.
MattGPT says:  so these 3 here, we know, are not useful columns.
MattGPT says: and then you kind of work through these right? And then the classification is sort of location type.
MattGPT says: a dress type. City. Wonder what city, you know? Does city have any variation in it?
MattGPT says: Yeah. Okay, so city is a sub.
MattGPT says: You can see like that's useful and interesting, right? But so then, what we're seeing is like
MattGPT says: location.
MattGPT says: We had borrow
MattGPT says: city  a
MattGPT says: and then you want to go through for those
MattGPT says: that's coordinating by board member
MattGPT says: grouping order together. So there's a lot of sort of overlap in that.
MattGPT says: And then we've got time.
MattGPT says: find and move.
MattGPT says: We've got created data.
MattGPT says: Resolution on day to day.
MattGPT says: Wow.
MattGPT says: yeah. So that's kind of basically taking the 2 sort of things seem to be like location at the time, and that
MattGPT says: okay.
MattGPT says:  And so you'd want to move them in that way. And that's what we're doing in Step 4 here.  and then you've identified your key columns. Okay, so we've got these 2 sort of things.
MattGPT says: We look at borough. We did that already. Right? You've got a distribution by borough.
MattGPT says: number are 2. Everything in this data set seems to be categorical, right. There's nothing continuous that's measured. It's not like the age of the person reporting.
MattGPT says: Or I guess, like we can, we could create some continuous ones like from the date and stuff. But here, basically, this gives us
MattGPT says:  the counts the number of rows there.
MattGPT says: Okay, so that would be what I'd do with like Bara. you could do the same with city
MattGPT says: alright.
MattGPT says: And what was one that was in there?
MattGPT says: Okay, so that overlaps with
MattGPT says: Fara, right?
MattGPT says: So
MattGPT says: say, like, Oh, yeah. these 2 are the same. So we can just get rid of one of them. Signal one of them
MattGPT says: and then
MattGPT says: city.
MattGPT says: There's a lot of values in there, right? There's a hundred 2 different options.
MattGPT says:  but again.
MattGPT says: they sort of overlap
MattGPT says: some way. So we've wanted to realize, like, what are the
MattGPT says: what? What are the, what's the the long tail of these? Right? So
MattGPT says: let's look at the bottom. 50
MattGPT says: get a sense of like
MattGPT says: what's going on there. And this is again
MattGPT says: messing public data. Mr. Brooklyn pose in there. Bedstein set up that Sliveson probably in that
MattGPT says: new high pop in lower case or title pace. When it's it's new high pop
MattGPT says: in Upper case those are the same faces there.
MattGPT says:  another misspelling of Brooklyn
MattGPT says: and the
MattGPT says: Brooklyn. Yeah, Brooklyn, New York, and so on. Okay, so
MattGPT says: look, you know, I would look at this.
MattGPT says: I would look at like this variable and say, it's not really that useful at the moment. It's gonna need a lot of work
MattGPT says: to actually do anything with it.
MattGPT says: right? Because what is it doing? Well, it looks like it's, you know, a free entry sort of
MattGPT says: thing. So when people report to 3, 1 one, or you know when the person that you call, and then someone is entering this into the database.
MattGPT says: There is a the city field.
MattGPT says: and people will kind of.
MattGPT says: Sometimes there is to think about where they are. But other times it just gets touched into sort of menu
MattGPT says: the Bronx. Right?
MattGPT says: so when you have a categorical variable like this. let's look at the top. Yeah.
MattGPT says: you'd look at it and see, like, okay, so what? What's going on here? Right? I think intention is to be much more specific than Barbara. Okay, so you're saying, I mean.
MattGPT says: I mean queens will wear in Queensight, in Astoria.
MattGPT says: flushing, and so on. But often it seems like you know. and the most frequent ones are Brooklyn, which is overlapping with borough. And then New York, which is all of the boroughs. Right? So it's going out
MattGPT says: and so on, right?
MattGPT says: And so there would be a bunch of work, probably, that you would need to do
MattGPT says: to make use of this one. And then, you know, fixing typos.
MattGPT says: fixing case and stuff like that.
MattGPT says: So that would be sort of my note and observation on that one
MattGPT says: would zip codes be better than yeah. So let's have a look at zip codes. Bye.
MattGPT says: does that have the codes in that? It does. It does. Okay?
MattGPT says: Hmm.
MattGPT says: it's called incidence. Zip. Oh, incident, zip, thanks.
MattGPT says: And so that's it. There it is. Okay.
MattGPT says: Aye. and accounts.
MattGPT says: Okay. So when you do that, right? So
MattGPT says: is it poses a good example. Let me just tell the zoom
MattGPT says: So I guess.
MattGPT says: Yeah. So with this right now, this is a categorical variable. But zip codes are numeric.
MattGPT says: Okay? And so we've seen this before is, if we just look at this
MattGPT says: we just look at this column here
MattGPT says: when you pick on here. It's probably the number of
MattGPT says: road. So we have it for everything. And then the D type is the data type. And remember going back to our pipe and basics most of the time. We don't
MattGPT says: have to worry like what type of object this is. The pipe will just do that sort of work for us.
MattGPT says:  but if we did account on this right for the number of
MattGPT says:  the number of rows that have values in, you can see that there are some rows where it's missing.
MattGPT says: right? And so
MattGPT says: is that I,
MattGPT says: someone's the other way that we could get at the number of missing. Okay. So we've got like a small, really small number. But there are 444 rows where there's no zip code recorded. And so what that will mean is in the data in the Csv file that we're putting display.
MattGPT says: And so the way the way the plan is working is like
MattGPT says: amazing one.
MattGPT says: one of those. When it reads a comment like that, then it will say, like, Oh, this looks like a number. And so I'm reading. There's a number, and then it hits a row, and it's like, Oh.
MattGPT says: there's a missing value.
MattGPT says: It turns it into a floating point number and a floating point where you get the bundle passes
MattGPT says: precedent because floating point numbers. for some reason compared to integers.
MattGPT says: can have missing values.
MattGPT says: Okay? So
MattGPT says:  that makes it a little bit tricky to to sort of to use in that way.
MattGPT says: And so what we'd have to do is think about how to best like
MattGPT says: deal with that issue. Right? So what I would probably do would be like.  is an a right. So find the rows are missing.  man
MattGPT says: missing, Zip. Okay, so that's the filter, right? And then.
MattGPT says:  thanks.
MattGPT says: Three-day approach.
MattGPT says: Was that 2?
MattGPT says: Then I want ones that don't have a missing. Right? So remember, you can invert a filter.
MattGPT says: And so now we have all the rows where there is a Zip code recorded.
MattGPT says: Feel forward? Answer than, Zip.
MattGPT says: okay, and then you can do something like as type
MattGPT says: equals.
MattGPT says: and I'm trying to enter an integer.
MattGPT says: And so you can see that now this looks much more like a bit
MattGPT says: same time. We don't really want it as a as an integer, because it's we're not going to add it up. We're not going to take a mean zip code.
MattGPT says:  so we actually want it as a string.
MattGPT says: Yeah.
MattGPT says:  That's why we have to turn it into an integer
MattGPT says: and then probably turn it into it.
MattGPT says: Strength. So these are some of the really gnarly like
MattGPT says: funds.
MattGPT says: Yeah.
MattGPT says: But like, I would do something like that something, you know. Sometimes you do have to take care of this kind of stuff.
MattGPT says: and then
MattGPT says: let's make a new column and call it like Zip code.
MattGPT says: And now we have something which is maybe
MattGPT says: every time we look at it we don't have to worry about. Why is there a point 0 after every time I'm looking at zip code.
MattGPT says: Okay? So then we get something like that. And we can say
MattGPT says: we're still getting 207 different. Zip codes tells us something about that.
MattGPT says:  but you know, let's take
MattGPT says: the most frequent ones
MattGPT says: that then gives us a way to sort of like
MattGPT says:  get a sense of like where those things are. This would be useful with some mapping. When we talk about mapping is that you can take the Zip code and you can find for the New York area like
MattGPT says: a map which is broken down by zip code. And then you could create a map with these sorts of things in there.
MattGPT says: Okay,
MattGPT says: so that deals with Zip.
MattGPT says: so we're not gonna get to those other notebooks.  about. Like, if I wanted to take?
MattGPT says: Yeah.
MattGPT says: specifically. And then within Brooklyn, look at the location text for
MattGPT says:  yeah. So you do, you have like, create a procm fail pack.
MattGPT says: Okay? So let's work with
MattGPT says: Well, we can go back to our Df. Too. Right
MattGPT says: And then it was borough.
MattGPT says: Percy
MattGPT says: Barra seems to be the one. Remember, we like city was this one that was like, we think, a free form entry
MattGPT says: sort of thing like, whereas I think, borough, when we looked at Borough.
MattGPT says: it's just got like the 5 boroughs plus
MattGPT says: unspecified as a as a label in there. Right? So this is probably a pull down.
MattGPT says: If you think about how this data was recorded and who recorded it. They've probably had a select pulldown, whereas this one
MattGPT says: must have been an open text field because we get things like Verona Rona right now for them. Right? So this would be less.
MattGPT says: but the less reliable as as a measure of that. So that's one of the things that you're trying to figure out when you look at the data there.
MattGPT says: So yeah, then we'd say, Barra
MattGPT says: equals. And it's all up the case right.
MattGPT says: Brooklyn, that would define your filter right? And then you could say, like.
MattGPT says: Brooklyn, df.
MattGPT says: okay, so that's subset rat. Df, 2 df. giving us 83,794
MattGPT says: things which could match the discount we had back up here. Yeah. So it just got the rows that have Brooklyn in the borough.
MattGPT says: How long? And then with that
MattGPT says: you could do.
MattGPT says: or the location type.
MattGPT says: and then say.
MattGPT says: a
MattGPT says: maps propaganda. Sorry. I'm sorry. DF,
MattGPT says: okay, so
MattGPT says: that gives you the breakdown
MattGPT says: for that. So that's that would be like that would be the easiest way. And and getting to the third part of what we're saying, like, we're doing this exploration. And now we're starting to move into like, oh, this is an interesting question.
MattGPT says: If you have some idea about Brooklyn like, say, compared to queens. Then it would be like, you know. there's a lot of twin houses there, or there's a lot of multi-family compared to if you're looking in Manhattan, you're not gonna get like single family very often.
MattGPT says:  so you know you could, your question could be, what's the distribution, or what are the most? The top 5. Most common
MattGPT says:  locations for rap fighting to be reported in Brooklyn? Right? So that would be your question. And then this, the steps would be define and build us, that I just get broken and rows
MattGPT says: and then count up.
MattGPT says: I create a ranked table or a rank distribution of the different location types, and then take the top 5 right? And then you answered your question right?
MattGPT says: and you could do the same thing. You could, you could say, Oh, what about Manhattan? What about queens? And so on.
MattGPT says:  an extension of that would then be sort of like.
MattGPT says: All right, let's leave. Let's go.
MattGPT says: This is where we do our grouping that we've seen before. So we can say, group by.
MattGPT says:  Fara.
MattGPT says: okay.
MattGPT says: And then look at
MattGPT says: location type
MattGPT says: and then size.
MattGPT says: Hang on
MattGPT says: no size. Sorry about accounts.
MattGPT says: And then see, can we put a head on that
MattGPT says: on that, and
MattGPT says: looking good by birthplace.
MattGPT says: Alright.
MattGPT says: bye.
MattGPT says: yeah.
MattGPT says: Head is not. Gonna give us this gives us the top 5 overall
MattGPT says:  about, Inspector. What I'm wanting to try and get is like a top 5 for each borrowing one step
MattGPT says:  that helps.
MattGPT says: And
MattGPT says: oh, yeah, let's try and move.
MattGPT says: Understand?
MattGPT says: Okay? So this time to get to something that could be interesting, which would then be like for each of the 5 boroughs, plus unspecified.
MattGPT says: The counts of.
MattGPT says: you know, by each of those. Right? So if we're looking at Manhattan
MattGPT says:  that was the statement. Law.
MattGPT says: you might want to, you know. Think about that, and make sense of like where there are vacant lots.
MattGPT says:  some account. It's interesting.
MattGPT says: A daycare nursery.
MattGPT says: Yeah, no, no. So it's kind of like, it's an interesting way to group your data and start to answer sort of questions that way. Right? So
MattGPT says: 2 2 ways out, a simple way like, and why I open to the like question is specific to a borrower so like filter
MattGPT says: the subset
MattGPT says: and then count up your variables been prepped right?
MattGPT says: You can do that, but
MattGPT says: when you assign it like Borrow, which only has a small number. It's got these 5 values, plus it's unspecified. That's where we find.
MattGPT says: So remember what Bouvier does. It's a jewelrying which
MattGPT says: divide. Yes, yes. So I borrowed.
MattGPT says: propane. Right? What we did, what you do with a filter is that?
MattGPT says: Say, okay, being a new data frame superintendent.
MattGPT says:  where you've got multiple values group by actually.
MattGPT says: does that all in one go instead of, I mean, create something.
MattGPT says: I'm gonna work with this. I'm gonna work with this man. And I'm gonna work with this. And then what you add in subsequently to that
MattGPT says: it will apply to each of these here.
MattGPT says: Okay, so
MattGPT says: we've got a little bit overcomplicate it. Yeah. But like. we just wanted to say.
MattGPT says: group by
MattGPT says: Barra.
MattGPT says: and then
MattGPT says: location type
MattGPT says:  value counts
MattGPT says: that's going
MattGPT says: borough by borough and then selecting the location type.
MattGPT says: counting up
MattGPT says: I'll have to think about that and get back to there. But really well. I guess you can see.
MattGPT says: And this is.
MattGPT says: yeah.
MattGPT says: Wait. How did you get all of the
MattGPT says: yeah. So that's just giving me the top 50 rows out of this. My.
MattGPT says: what I'm trying to do is I really, wanna I'm really wanna be able to get the top one or the top 5 or something for each borough, and then compare them.
MattGPT says: Be a really good question. So and
MattGPT says: how do we do that?
MattGPT says: Have to think about that one.
MattGPT says: And then, like, in terms of asking. But email, the data. So
MattGPT says: I made like a plotted
MattGPT says: Brooklyn and Queens for location type. And
MattGPT says: like, they're a little different. So when would a question be
MattGPT says: like, why would
MattGPT says: they like? Why is more have more rest, sightings and family dwellings than
MattGPT says: program? And
MattGPT says: yeah. So the first question would be the comparison that you did right, which is like.
MattGPT says: you know
MattGPT says: one of this, you know how similar is
MattGPT says: book into queens in terms of the types of places like, I guess.
MattGPT says: And that's why I was trying to get to this top 5 like, what are the most? Or where are rats most? What type of property are? Are rats most frequently reported in
MattGPT says: in in queens compared to
MattGPT says: program, or something like that. That would be the question. I think right? So sort of the comparison.
MattGPT says: Now the why question kind of comes a little bit further down the line, because then you'd have to think about like.
MattGPT says: why is that right? And to do that, you'd have to go and find another data source right? You'd have to go and find like
MattGPT says: housing types by Bara in New York City. And then you could sort of come up with a normalization which is like.
MattGPT says: you know. Cause if you were doing that in, say, look at Manhattan. Well, what are most of the buildings? Right? Their apartment blocks.
MattGPT says: and so
MattGPT says: there are some brownstones and stuff like that. That might be counted as like
MattGPT says: 1. 2 family dwellings, I guess, but like.
MattGPT says: cause I would.
MattGPT says: Yeah, so like 3 plus family apartment building. Yes.
MattGPT says: Was that so where we had that?
MattGPT says: So let's say, we.
MattGPT says: yeah, I guess this is one way to sort of do it.
MattGPT says: I sort of turn that into
MattGPT says: when you got. Remember, you can sort values
MattGPT says: on a specific column, right? And so we could say.
MattGPT says:  And it always goes from smallest to
MattGPT says: largest.
MattGPT says: since it's sending people through. Now, it's sending each
MattGPT says: powers in
MattGPT says: phones.
MattGPT says: Okay, so
MattGPT says: now we're getting it like. yeah, 3 plus family apartment buildings.
MattGPT says:  yeah, that's I mean, it's there's a considerable amount, actually more than I would think.
MattGPT says: That's why the Manhattan bombings.
MattGPT says: And
MattGPT says: so going back to this one here. Another thing that we could do. Remember, you know, using our
MattGPT says: and
MattGPT says: aye.
MattGPT says: alright
MattGPT says: by consuming all that
MattGPT says: something.
MattGPT says:  we could try. I,
MattGPT says: Laura type.
MattGPT says: and
MattGPT says: we're only interested in like modules.
MattGPT says: you under signings. Maybe. Then
MattGPT says: same idea of like.
MattGPT says: okay. So now we're going to somewhere where we can do comparisons right and see that.
MattGPT says: So whenever you get a frequency table just like we had with 5 accounts you can do, you can do some filtering on it here.
MattGPT says: So here. let's let's be even more restrictive and say that like gonna be more than a thousand in each.
MattGPT says: And so the most. and that's getting us close to that top. 5 right?
MattGPT says: So the most frequent sighting in Brooklyn is in a free plus family apartment building. It's the same in the Bronx. It's the same in Manhattan.
MattGPT says: but in queens that doesn't even come. That's the second most frequent one
MattGPT says: on Staten Island. That's not
MattGPT says: their tool right?
MattGPT says: and then
MattGPT says: 1, 2. Family dwelling is the second most common in
MattGPT says: these 2.
MattGPT says: But, Manhattan, there's much less of those that make sense in queens. It's the most frequent, and Staten Island is the most frequent
MattGPT says: right? So that in a way, this is an indirect way of getting it like. what's the distribution of property types
MattGPT says: in in New York City by by borough. Right? But to go back to sort of your question of then the comparison. It would be like.
MattGPT says: yeah, we really need some other way of some other data source which says, like, 70% of Manhattan buildings are
MattGPT says: 3 plus family apartments. and then you could take the number of sightings and kind of scale it by that. See it?
MattGPT says: What's going on. But this I mean this in a way like a rat signing a rats don't
MattGPT says: distinguish between building types, and so they could turn up anywhere. So this is a sort of you could use it as a proxy to say, like
MattGPT says: rats, turn up everywhere and anywhere. And so by doing this sort of categorization and this sort of counts.
MattGPT says: This gives me an indirection sense. If I don't know New York City at all.
MattGPT says: What the property type distribution is maybe would be, you know, one way of getting at it
MattGPT says: cool. Alright? Well, it's time. So again. Sorry about the tech difficulties at the beginning, but hopefully.
MattGPT says: like you get the idea.
MattGPT says: the big things I would like to take away is like when you're working with real data.
MattGPT says: It's much more messy than the data set that I've given you in assignments. And so you have to do a lot of like. That's why
MattGPT says: we've got these steps that we're working through, which is like the investigation part is not just sort of like, oh, how many rows, how many columns? Oh, yeah, it's actually then, really drilling into the data.
MattGPT says: going through all of these stages. And so what we did here was sort of found out that like there's some columns that are. There's no variation at all like complaint, type.
MattGPT says: and agency and agency name. And so there's no point even looking at those once you understand that I mean, you look at them. But then you very quickly say.
MattGPT says: okay. So out of the 36 columns, we got down to a small, a much smaller subset and we saw that location is the main one that we can work with here.
MattGPT says: and then time. And so and then time is actually a little bit tricky, and we'll come back to that next week. Is that the only time variables here are these timestamps we've got created data.
MattGPT says: And then we built like an updated one. But here, inside this, we get like
MattGPT says: this is, it's actually rich because we get the gear. And we put the month because of the day and day of the week and time of day.
MattGPT says: And so if you do, look at the second notebook, and then you can just run all the cells in there.
MattGPT says:  and we'll come back to this next week, but like  you'll see
MattGPT says: a lot of this in here to actually make use of the time. Temporal variables.
MattGPT says: We want to split apart the created problem.  and
MattGPT says: out of that stream that gave us the year, the month today. And we're gonna no time today. But like you can get the year, the month of this week
MattGPT says: of the year, and today. To to be calculated right. And so then you can then have a new column, which is year and so over time.
MattGPT says: Since this database is being checked. Right then you can do a plot like this, which is so the number of reports of rat sightings has followed this trajectory.
MattGPT says: Okay? So it's an upward trend. There's the trend. But like there's something interesting going on in the story. Why did it dip in these this period of time? Pandemic, I think, from what I remember like, they stopped a lot of services during the pandemic, certainly at the beginning of that.
MattGPT says: But then you could say, Okay, so
MattGPT says: we're clapping across all of the years. Where? What does it look like over months here. So we've got the 12 months of the year
MattGPT says: group filing month. And then geography. You're gonna see more red signings in the top box right?  that's because people are outside, and that's probably because more rats come out when the weather's warmer.
MattGPT says: But it's more about people probably being outside, or I would, I would guess right.
MattGPT says: And then we could do something like this, which is okay. So over time.
MattGPT says: let's group by year and month. So we're getting a data point for 2010, 1, 2, 3, 4, and 12, like those of numbers at the month, and doing that all the way through.
MattGPT says: And this is what in the readings is we can talk. We talk about trends. Okay? So the number of signs at the beginning of the database and the number of signs. And so there is an trend.
MattGPT says: But the cycle. These are those annual cycles that we just saw, and they're repeating and repeating again. You could break that down and say, Okay.
MattGPT says: we looked at months of the year. What about weeks of the year? So instead of 12, like 52 points.
MattGPT says: get the same shape. But there's a little bit variation. Does that pay you anything extra? And then here you can combine that which is the year and the week.
MattGPT says: Okay, so we're just seeing this pattern here repeated again at the cycle.
MattGPT says: And then you go down to the day level, and as soon as you go down to the day level. It's just wait, is really noisy, right? So there's something going on in days of the week, maybe. But most I would look at that and say, like, it's not really anything. There's more noise in that than there is signal right? I can still just about see
MattGPT says: the cycles.
MattGPT says: But there's one data point in here that signed out
MattGPT says: some ally of some era. This plot can be useful for helping you find that be like, I've got to figure out what's going on in there. And so if you reach Chapter 5 and learning to see data, he talks about this particular data point and what it is.
MattGPT says: Okay. Thanks. Everyone. Sorry to go over time.
MattGPT says: Member quiz. It's posted due on Sunday and
MattGPT says: I'll see you on truth.
MattGPT says: Oh, not a leg.
MattGPT says: Okay, okay.
MattGPT says: Only time. I was like, just can't do the
MattGPT says: okay. Hi, everyone.
MattGPT says: I have one.
MattGPT says: yeah. So kind of
MattGPT says: a shameless plug for next semester. I'm teaching a new path which is all about generated AI communication.
MattGPT says: think about Jani from 2 perspectives, from a communication perspective. This wasn't. Why, what's amazing about Canada. AI. Why, everyone was certified about
MattGPT says: it's in a you know, we have a communicative bias as human beings, because we're social and we communicate. And that's how we and Brand, and we also have a conversational bias, because, you know, basic level of communication comes through conversation. So when you put a genai which is not new technology into a chat of 2 things, and you communicate in that way. Then it's something like Whoa
MattGPT says: with mid twentieth century, which is sort of like a test. Where?
MattGPT says: Imagine.
MattGPT says: you know, behind the screen you pop bits of paper underneath and back and you can't tell
MattGPT says: whether you're talking to a human being or
MattGPT says: machine, right? And so
MattGPT says: it's amazing what it what it gives out.
MattGPT says: Then we have this sort of bias that we then assuming stuff. So any of you that have played around with it and started doing prompt prompting.
MattGPT says: You know, I'll see people doing prompting and do things like you are a such and such. You are an intelligent agent, or you are.
MattGPT says: And so there's this personalization of it, as well, you know. And I am this building this relationship. So there's a lot of really fascinating stuff to think at
MattGPT says: with, to think about and talk about generative AI from communication perspective areas of communication
MattGPT says: Apis
MattGPT says: to do some little bit by going back to some of the older models to see exactly how they work.
MattGPT says: So I'm really excited about that class. That's the kind of stuff that we're going to do. So I'm happy to talk to you more of you interested, but I'd love to have any, all of you in that class.
MattGPT says: Yeah. This is like partially a tangent. I was just wondering if you saw like. I saw this thing in the news like yesterday, maybe 2 days ago. It's called like night shade. It's like a data poisoning for like
MattGPT says: for like like the where they call the Lms or whatever. Yeah. So it's like, basically like.
MattGPT says: if you ask for it to like, make an image of a dog if it scrutes.
MattGPT says: or if it scrapes an image off the web. That is one of the like.
MattGPT says: That is like data poison, so to speak. It won't like it won't be able to generate like images of like a dog correctly anymore. I mean, II could probably send you because I'm totally explaining it. But yeah, I know I know the kind of thing right? And this is kind of like.
MattGPT says: one of my packs, and there's a whole series of stuff up or and you make models for the gap.
MattGPT says: Let's come to a quote on that. So yeah, it's interesting.
MattGPT says: But yeah, if you can send me the link down.
MattGPT says: So yeah, just wanted to let people know about that kind of popular process.
MattGPT says: what we're going to do today. It's got a little bit. I just want to do a little bit more about scraping but mainly just
MattGPT says:  Look at the project setup. Okay, so everyone, if everyone can check that home folder
MattGPT says: on Jupiter hub.
MattGPT says:  you should see
MattGPT says: like it's if you actually go to the home folder. So not you know the the the main the main base should have a folder in there. 31 80 underscore data project group. And right? And that group end should be
MattGPT says: the group that you're in. So I'm hoping that I got that right. Right? So
MattGPT says:  that will.
MattGPT says: okay. So you should see. And I I can see all of the projects. Right? So there's 9 groups. So same group number that you have on canvas 9 groups. You should have the right version of this border
MattGPT says: in your home level. Does everyone see that should we should be just at this top level, so go to the folder. So not inside.
MattGPT says: But you know not not inside from 31, 80, at 23, but actually right at our level, it will be equal with that. Okay? For you, at least. Right? You know, Project Group A,
MattGPT says: so yeah, right? Got it.
MattGPT says: And so if you look at that, what that would then is is this the link shared link so that everyone in the group can access is an IP to
MattGPT says: this is phone off Github. This is the template for this. So we've looked at it before in the previous project.
MattGPT says: Hey? A data folder data analysis. presentations, data story outputs. Okay?
MattGPT says: so this will be where you'll you'll you'll be working. And you can start working right now. So
MattGPT says: you know, you put stuff, put stuff in the data folder, and everyone in your group will be the data that
MattGPT says: and then I can also jump in and like, if you got questions or you want to help with stuff, I can put a notebook into your data analysis, holder, and help you get started with stuff, and so on. Okay, so this is the way this is how you're gonna collaborate and work on your data project
MattGPT says: within your group
MattGPT says: works. Reasonably. Well, okay, it's not. There's some limitations to the way that this. how the Jupiter works is that it doesn't have true collaboration. In a sense, we we got
MattGPT says: where, you know, 2 people are open at the same time.
MattGPT says: And
MattGPT says: let's do this as an example. So this is Group 9, which is Darren and your group? Right? Okay? So if I start a new file here? Well, actually, no, that's one thing to check file 2. Is that book?
MattGPT says:  okay. So I saw, and a new notebook in here.
MattGPT says: Can you open up that one that I just personally name it. So that's yeah, let me just
MattGPT says: sharing.
MattGPT says: Okay? So
MattGPT says: I'm working. If you open up right you see it right? So if you
MattGPT says: so you'll be able to see it. So I've just created, and a book that's down. And then the other
MattGPT says: person in this group can see it right. Shelby will be that person. But if you open up right. But I don't see like the update, just like, yeah, exactly. Right. So now, if you.
MattGPT says: So let's just wait a little bit. So can you put something into that? If you open up this file? Yeah, just my panel put in one plus one or something like that.
MattGPT says: Yeah, so okay.
MattGPT says:  so we don't. We don't have true collaboration in the sense that, like you're looking at the vendor. But what I'm looking at, but like.
MattGPT says: so you just have to be a bit careful with that. So basically like one, you can see notebooks that I was working on bone and one, but really work at it at the same time, and otherwise you'll get complex.
MattGPT says: But what's gonna happen here
MattGPT says: if I save this end.  see? Me?
MattGPT says: Yeah. So that's that's the the limitation to it, like overall, though, it has to work recently. Well, okay, so that's how you can start to get working on your on your project, you can do all of your
MattGPT says: that's part of what we do, all of your analysis.
MattGPT says: put the data on here with a set of makes it much easier for me to collaborate and interact with some of that.
MattGPT says: Okay, so the data folder is where you'll have data
MattGPT says: and within that you can create substructures. So often you'll have your raw data. But maybe you've downloaded from somewhere like survivor team. Then you've got your audio files. You can put them into a code there, and then you can figure out the conversion right? So there's cup on the server, you'll find there's like libraries that might help you. Our data
MattGPT says: occupy.
MattGPT says: And the only thing that you can do is that if you want. You can start a notebook, which is an hour notebook right since you'll notice here that the on the server you can have a notebook has a behind it.
MattGPT says: They don't, really. They don't want you to do the whole analysis enough, but certainly openly. You can certainly open the files there and then save them out.
MattGPT says: Why would you want to be the okay, is it
MattGPT says: possible to like create 3 separate notebooks? And then all of this live work, you know, have like one notebook, that kind of puts it like, cleans it all together. Yeah, and gets into like a final date. And then
MattGPT says: like, load that into 3 separate notebooks. And I do my analysis exactly right. And then we can like copy these together. That's should that like work data analysis. You want to have some net books right? And then we'll want the number for the whole. Okay? So
MattGPT says: so you can have them.
MattGPT says: And then just close it down when you're done with it, and then Sam can open it, or if you can open it and look at it and get around that sort of process, right?
MattGPT says: So it's not perfect. But in in reality there's very few problems that you have, right? It won't close down the notebook there. But yeah, you want to divide your cost up in the Channel spreadsheet.
MattGPT says: Different people click on and then have a note of each vote.
MattGPT says: Okay? So if you look at the previous projects and data analysis, you'll see there's like data cleaning initial data analysis data set, one initial data analysis of data, set 2,
MattGPT says: and then at some point, you know, you will start to go to a place where there is a main notebook that has
MattGPT says: more detailed analysis like in your case like you. But we'll do something
MattGPT says: like one of you may take on the building social network out of layers or something like that. So
MattGPT says: they're on the side of more notebooks than less right. But you don't want big notebooks that have low, level and stuff, and you can have a sort of working one. But you really want to have them really nicely documented and build that structure.
MattGPT says: Someone can come into your project and be like, Oh, okay, I understand this notebook is like cleaning. This notebook is exploration. This notebook is
MattGPT says:  questions, these questions, building social networks. This one is like
MattGPT says: patent finding, or whatever. You know, we should have a way, I'm not gonna tell you how to break up your notebook.
MattGPT says:  okay, so that's the
MattGPT says: that's data analysis. Okay? So all the notebooks, basically that go from the whole stage of acquiring your data. If you're doing so, scraping or downloading or cleaning your data, if you already have some data, but you want to then subset it, break it up.
MattGPT says:  and so on goes in your data analysis, right? Presentations that just replace the 2 presentations for the project. Right? There is the initial one that you're going to do a week today. Okay, so 3 to 5
MattGPT says: minutes.
MattGPT says: 3 to 5 slides. That introduce? What is your research question?
MattGPT says: What day are you working on? What's your planned analysis? Okay? So you can put that across 3 to 5 Slide.
MattGPT says: You can decide how in your group you all want to say a little bit, but you want to designate one person to do it. but we'll go through each of the groups next Thursday, and you can give your presentations.
MattGPT says: In that way. Okay. And then we'll get feedback from groups. And then in the last week of semester, then you'll do your final presentation.
MattGPT says: which is at that point. There's still a week after that 3 you actually have to submit the thing. So that's like where you've got to so far. So you've got like 10 min per group. and we'll split that across those 2 last classes.
MattGPT says:  we talk more about the kind of final presentation. Okay.
MattGPT says:  and then every group we'll have one data story, right? Which goes in the data story
MattGPT says: hold up. And that's basically what you've been practicing. So the blog post sort of idea.
MattGPT says: kind of thing that you'd find on medium com, which is a self contained engaging data story from what you've done. Right so.
MattGPT says: And then we sort of landed
MattGPT says: 15 min read, depending on how you want to do it.
MattGPT says: Okay? So there's one of those for each group, and you can collaborate on that. And then what's new that you won't see in the previous periods is that I also want you to think about the creative output
MattGPT says: alongside
MattGPT says: the data storage. And for that you can do what everyone. Okay. But that could be. If you want to make it.
MattGPT says: you wanna take the easy cost that would be like, create a poster right?
MattGPT says: a presentation post that
MattGPT says: and put that together like in a powerful.
MattGPT says: you want to do a video where you sort of like. yeah, the Youtube style kind of thing you can do that.
MattGPT says:  if you want to create a dashboard
MattGPT says: that is interactive
MattGPT says: supports your data. Sorry. Do that.  When I create a board game.
MattGPT says: write a poem. be a song.
MattGPT says: do, and interpret it on, and get some sort of creative output alongside your data, solving right? So this gives you a chance. The things you can find
MattGPT says: do some sort of additional output.
MattGPT says: Okay.
MattGPT says: any any specific questions?
MattGPT says: No. Okay.
MattGPT says:  let me just
MattGPT says: how many people.
MattGPT says: so I wanted to do a little bit more on web speaking, and then we can spend the rest of the time you can work on your project and I'll come around and help people.
MattGPT says: How many people are familiar with the Wikipedia philosophy game
MattGPT says: people heard of this? Okay, so if you go to Wikipedia
MattGPT says: and put in any topic. Right? So let's put in like data
MattGPT says: the book that I can't remember who came up with it, or I've read some and came up with it. But how the game works is. You go to any Wikipedia page.
MattGPT says: You ignore the defense first paragraph, which isn't ambiguation, or like links or sidebar, or anything on that.
MattGPT says: find that first link which goes to another wiki entry and then click on that. Okay? So we go to data.
MattGPT says: the usage.
MattGPT says: and then you repeat, you go to language.
MattGPT says: and then you go to communication. and then you go to information.
MattGPT says: and then you go to abstract concept. And the the argument is that you keep doing this. It takes N steps, but you will always, or most of the time. Then we go to rules. Get to philosophy.
MattGPT says: philosophy of logic. We're almost there, and there we are, block
MattGPT says: so good. If you want to try that like, let's try like negatively.
MattGPT says: oh, you'll always get to philosophy. Well, yeah, that's the argument. Right? So an egg would go to fertilize.
MattGPT says: Yeah, this, I think this does count. I'm not sure that what the rules are, but like, we gotta if we got English dialects mutually comprehensible linguistics. Okay, we're gonna get one right like
MattGPT says: communicate that.
MattGPT says: But that's the claim, right? That like, I think it's 95% of the science you'll you'll get that right.
MattGPT says: I mean, I'm concerned, too. So
MattGPT says: so try try some people, too, like Try Britney Spears, or something like that. Right? Ii got it.
MattGPT says: But you have time. So what what do you think like. Well, can you think? Figure out something that takes a while to get there?
MattGPT says: I don't really know. Kind of what this phenomenon is right.
MattGPT says: Oh, no. First thing is different.
MattGPT says: and you hit the velocity one. I would think that I would get the. And then you go to simplify Chinese.
MattGPT says: Okay, yeah. And then you go to Chinese languages.
MattGPT says: Yeah, okay, so yeah, there are some loops. Right? That's an interesting one. Another one that you'll find is a loop, which is surprising. It's psychology. You think the psychology would get away right.
MattGPT says: But psychology goes the money
MattGPT says: goes to.
MattGPT says: I think, things. I think I think psycho teams at least
MattGPT says: punches. If it connects to language, it's over awareness
MattGPT says: psychology and then moves back around again. Okay, so something interesting about psychology. Yeah. So it's kind of self-referencing. The next link over from that was like, was philosophy. So it gets in there.
MattGPT says: Alright so kind of fun fun thing. You can try that out.
MattGPT says: I'm not sure I'm not sure why it is, you know. I think it's the encyclopedia thing, and at some point you've got definitions at the top. And so you're always going to get back at this kind of like a
MattGPT says: a knowledge graph. Or, you know, a tree right? Sort of a hierarchy of stuff.
MattGPT says: But if you want to if you want to grab border 14  then
MattGPT says: what I want to want to show you is just like, how can we implement that right? So that we don't have to
MattGPT says: click? So if you go to folder 14
MattGPT says:  and take a look at this Wikipedia game.
MattGPT says: There's some links there about it. Oh, there's the actual Wikipedia Page about getting to philosophy.
MattGPT says: Wonder whether that one actually
MattGPT says: Lutheran itself. It should be clicking
MattGPT says: you to use a
MattGPT says: computer. Yeah, computer will get us that right machine
MattGPT says: power
MattGPT says: physics. Okay, where that as soon as you get to a like a
MattGPT says: the discipline. You're gonna get the branches of clients. And then we're gonna get to like scientists.
MattGPT says: systematic
MattGPT says: empirical members. proposition.
MattGPT says: philosophy of language, philosophy. So analytic philosophy.
MattGPT says: tradition.
MattGPT says: So okay, that mainly broke it. Anyway. The point is, you know.
MattGPT says: you can play around with this. But what I wanted to would show you is that we mentioned this sort of at the end of of class.
MattGPT says: It's a way to sort of it's to automate this right? So let's say
MattGPT says: stuff with egg right?
MattGPT says: Some of you, maybe some of it for some of you, hadn't we? But laughing.
MattGPT says: But remember, if you are on any web page, and you click, inspect.
MattGPT says: Then the browser. You'll get the the inspector
MattGPT says: which gives you a navigate, you know. A nested version of the HTML constructed markup language
MattGPT says: behind the web page. Right?
MattGPT says: Wikipedia is nice because it's relatively simple and web-weight web pages go these days because it's spying and pretty minimal.
MattGPT says: But
MattGPT says: the way the HTML work is you've got these nested tags or elements. An element is defined by the angle bracket open
MattGPT says: an element name
MattGPT says: series of the attribute values
MattGPT says:  are on the open element.
MattGPT says: And then we have. That's the open element.
MattGPT says: And then the clone. It's just the
MattGPT says: less than with flat. And then saying, How do you think?
MattGPT says: Yeah.
MattGPT says: And it's a hierarchy. So then, elements can be nested inside each. However.
MattGPT says: And then between the angle brackets. No, inside. An element is where you have function.
MattGPT says: So that's how a webpage works. Right? So HTML is the top level one that's called the root element.
MattGPT says: and then inside that, usually you have a head and a body. The head is where you have a lot of spec. You know. You specify where title is, and then
MattGPT says: where a lot of stuff goes these days, and
MattGPT says: loading in style sheets and loading in javascript etc., the body gives you the structure of the
MattGPT says:  And then you can see these attributes on here, wherever you see a class that's usually related to some sort of like styling
MattGPT says: So when you're doing web scraping, and you're trying to get stuff out of the page, understanding you're always gonna need to look this, understand the structure of the page
MattGPT says: and try and find like, okay, I'm interested in content where the light so you can see here. Div stands for the division
MattGPT says: generic way of providing things up.
MattGPT says: And then we've got class
MattGPT says: MW. That page, Dana.
MattGPT says: And so you can see, as we navigate around. and
MattGPT says: we can try and find
MattGPT says: the main part of that without the table of contents.
MattGPT says: So you can go in that direction. Sort of understanding the structure. You can also say, Oh, okay. What I'm really interested in is it's first link right here.
MattGPT says: So if you right click on on that link, and then you'll find it right here.
MattGPT says: Wiki fertilization. It's in an a element which is a hyperlink. And you can see that it's inside this paragraph here.
MattGPT says: Okay, so that's the first paragraph. So what we're doing when web scraping is all going backwards from
MattGPT says: what we, you know, the element that we know. And we're trying to find a most reliable and least convoluted way to locate it in the HDMI
MattGPT says: in the structure. Yes.
MattGPT says: so we're looking for a P. Element. Stand for a paragraph. This one is just over one
MattGPT says: but that's the one that we want. And then inside that, once we find this element.
MattGPT says: the way the game rules seem to work is that we then got to find the first one of these the first link here.
MattGPT says: Now there's a bunch of links up here, too, that we can see here. But this is, it says, ignore the disambiguation stuff, and nor any sort of
MattGPT says: quote boxes stuff like that. Okay?
MattGPT says: So web escaping. Oh, you know, walking around is a little bit of a dark
MattGPT says: start up, and it's it's fun to do. But you can waste a little bit of fun doing it.
MattGPT says:  but this this is kind of as it's most basic is that
MattGPT says: you've got 2 steps. The first thing you need to do.
MattGPT says: and
MattGPT says: get the content of the page right?
MattGPT says: And so that's doing an Http price.
MattGPT says: And then
MattGPT says: you wanna
MattGPT says: pause upon them
MattGPT says:  and but I'll deck
MattGPT says: and then
MattGPT says: navigate correct
MattGPT says: data on that.
MattGPT says: And so
MattGPT says: there's different ways to do it, and pipeline and in other languages. But the request module, you'll find, is still the one that will send you the most.
MattGPT says: the really lightweight way to get at
MattGPT says: what you do in a browser getting a URL and getting the content back.
MattGPT says: And then you also need some way to do this second step, which is what you'll get back when you say Yes.
MattGPT says: wikipedia.org slash a wiki slash a
MattGPT says: is that you'll just get everything you'll get back all of the HTML.
MattGPT says: What we would just looking at that. Okay.
MattGPT says:  So let's do
MattGPT says: do our a example here.
MattGPT says:  so we'll come back to this in a second. But let me show you the. So the first thing you do is
MattGPT says: use the requests.
MattGPT says: Say, yeah.
MattGPT says: then let's say, our base. Yeah, URL,
MattGPT says: and
MattGPT says: Okay, so that's how you do step one. And then you look at the response to that.
MattGPT says: See that it's going up.
MattGPT says: Requested that page, and if you get back to 200 level. That means it's a 200 service code. Then that means that you
MattGPT says: got some content. Now.
MattGPT says: a
MattGPT says: gonna give you back a big lot of strength.
MattGPT says: which is the string object. Put the content in there. So
MattGPT says: it's you know. You could then just play around this, and you could try and so do like searching and string, substring to try and find the first P.
MattGPT says: But because of the nested nature of HTML. Elements contain other elements which contain other elements. You actually.
MattGPT says: you know, it's it's gonna be really tricky to do it just with string handling.
MattGPT says: But what you tend to do is
MattGPT says: pass it into an object. Okay, it's a beautiful soup. It's a really old module, but still used
MattGPT says: a lot, because it's a good one.
MattGPT says:  And so what we're doing here is that red dot text is the stream content and the stream object of what we got back from
MattGPT says: going to Wikipedia. Dot org's like Wiki's like, hey? And then we're passing it to beautiful soup, which turns it into beautiful seat. Okay? And so
MattGPT says: a
MattGPT says: you can see it looks like we're just getting a representation of it. But this is aware of the structure of the page.
MattGPT says: And so you can do things like
MattGPT says: fine. P. Okay, so remember, object is a thing that holds data, and it has
MattGPT says: functions that operate on that. So there are
MattGPT says: functions to find specific elements and elements with specific attributes and etc. That's there's a whole lot of something, right? Yeah.
MattGPT says: I found something that doesn't go to philosophy
MattGPT says: breaking bad help to do. It ends on a loop on consciousness and thought.
MattGPT says: That's it so close. So you can see that like, you know, you can play around sort of with this. That's fine. There's fine
MattGPT says:  that will give you a list of each of these. So you can see here, we've got a list of
MattGPT says: development objects.
MattGPT says: and the first one we don't want. We want with that one. But it's not always necessarily going to be the second one. So we need to think of some other ways.
MattGPT says: Locate that. So that's the basic sort of idea behind this whole approach. You see, this is found when you say? Fine, all of the find you all of the P. Elements in the whole dust.
MattGPT says: But this function is really, I mean but I play around with it a little bit.
MattGPT says: It will implement all of these things, and what it does is first
MattGPT says: the page that you, the Wikipedia page. You pass it, it, get, go and get it gets that turns out into HTML string.
MattGPT says: fathers that. And then it's looking for this
MattGPT says: thing called Div. And MW. Content text that then inside that has another div that has. And MW. Partha outfit.
MattGPT says: Okay? And that's our main div there. So if we look here.
MattGPT says: the not that that will be that one right there.
MattGPT says: Okay, so and MW Funden text. And then you can see the event within that we've got. And MW. Barter quote
MattGPT says:  and that takes us. We still then have to try and find
MattGPT says: the first paragraph inside that.
MattGPT says: and so we don't want a paragraph that has a thoughts attribute that says, Enemy.
MattGPT says: and we don't want a paragraph which is inside. We want to paragraph which is directly a child of this element here.
MattGPT says: Okay, so
MattGPT says: that's what all of this kind of stuff's doing.
MattGPT says: And then it does it recursively from wherever you start.
MattGPT says: Better run it better.
MattGPT says: Okay? And it does the loop around that
MattGPT says: seems to get messed up.
MattGPT says: Okay, so
MattGPT says: is it breaking underscore that? Yeah.
MattGPT says: that will lose. and another one if we want.
MattGPT says: I don't think a show would do that
MattGPT says: interesting. Okay? So then, as as soon as it gets to psychology, we know that it's gonna loop, right? Yeah. And then I just kind of built in.
MattGPT says: you know, if it if it gets back to the same people 3 times and stop.
MattGPT says: I'm not sure. Romance languages.
MattGPT says: Oh, okay, yeah. I was clicking the it may be clicking the wrong one, says Wikipedia. Please clarify.
MattGPT says: So we maybe would need to sort of add in some new rules, which is like.
MattGPT says: ignore anything that has like a Coulomb in it, as the last thing here, and that we maybe should go to reference first.
MattGPT says: but it looks like well, not that, you know. I guess we'd go. What would happen if we were here
MattGPT says:  modern languages, I guess where we go.
MattGPT says: And then we're gonna get back. Yeah. So it's kind of a waste.
MattGPT says:  you know. Why? So yeah, that's something about like something on that particular page where it's actually locating something.
MattGPT says: Physics.
MattGPT says: This, the trick is actually harder than you think to find the first parallel.
MattGPT says: That isn't all of these other things, just like there's all sorts of stuff. There's a disambiguation. There's sometimes like pronunciation, and sometimes a little box of stuff like that.
MattGPT says: So
MattGPT says: yeah. And people want to play around with it and see if they can find a better
MattGPT says: way to locate the first paragraph.
MattGPT says:  okay, so it's a, you know, pseudo, somewhat fun.
MattGPT says: kind of game or phenomena. But the the real point I just want to show you is like you can
MattGPT says: with a little bit of both
MattGPT says: and underlying the structure of a web page. You can actually automate that. And some people have done that
MattGPT says: in a more robust way that doesn't break
MattGPT says: and found, you know, filled up sort of piles of stuff like this.
MattGPT says: Okay? So  try like beer.
MattGPT says: So dictionary.
MattGPT says: I guess, or you, you know, you could come up with a list.
MattGPT says: What would be interesting would be, what is the distribution of status like
MattGPT says: and like it was just saying, like.
MattGPT says: What did you?
MattGPT says: What did yeah. Like something like Britney spears?
MattGPT says: Oh.
MattGPT says: or other celebrities?
MattGPT says: Guess that remarkably quickly.
MattGPT says: So how do you 14 establishes? And so you go through this kind of pop?
MattGPT says: And then you're at some point you're gonna get to something like language. And as soon as you hit language, you're gonna hit communication. And back, you know, you're up there.
MattGPT says: Yeah,
MattGPT says: the instruct mandolin breaks it.
MattGPT says: but breaks it. Yeah, yeah.
MattGPT says: yeah, that's probably, you know. Try it. Try it doing it like manually to see like, probably we'll get there. It's probably just there's something in one of.
MattGPT says: So I'm not giving you this solution that will always work right? It's not shipable to
MattGPT says:  but it illustrates sort of how you can do stuff with with web straightening
MattGPT says: and collecting data.
MattGPT says: by understanding the structure of the page.
MattGPT says: Following those things that load the page.
MattGPT says: pause the page, find the content that you want and totally extract that
MattGPT says:  The other notebook in here just does the same thing.
MattGPT says: But,
MattGPT says: we're going to look at the passengers on the Titanic data set map, so you may have looked at it on the board like
MattGPT says: on haggle, and a
MattGPT says: maybe next week, when we I want to look at the
MattGPT says: predicting, using staff models and socket, learn
MattGPT says: a building model.
MattGPT says: And so the Titanic passenger list dataset has been used as an example here, where you've got like these are the passengers that age and where they audit and what class they're in. So they're first pass, and then whether they provided them off.
MattGPT says: Okay, so there's a whole load of stuff there. So can you predict from
MattGPT says: any of these speeches?
MattGPT says: who is likely to
MattGPT says: Harrison, and who would like to survive.
MattGPT says: But this is an example where actually. you can get.
MattGPT says: yeah? So yeah, yeah, they hit some of them. One of the roads there, and you see those lights
MattGPT says: plus seems to be. It's not great visualization, but the the right hand side of every sort of thing. Here people in the right thing are victims of these as the dark ones and survivors.
MattGPT says: the crew, the majority of the crew well and then gender.
MattGPT says: First class women pretty much all survived.
MattGPT says: So these are the patents sort of in the data.
MattGPT says: So if you were interested in sort of working with that, then.
MattGPT says: And data set does exist in Csv, and so we'll use that. But imagine that it didn't. And you're like, oh, this could be an interesting sort of problem.
MattGPT says: some point. Those will the data make the way of video or websites where you have to.
MattGPT says: And so how can we get the data out of there.
MattGPT says: Well, you you want to go for the path of least resistance.
MattGPT says: And so we've seen this already.
MattGPT says: Is that in Pandas
MattGPT says: you can use the read HTML, and
MattGPT says: that will give you back a list of all the tables on the page. Okay, so there.
MattGPT says: there are 12 tables that it found on this Wikipedia page below, let's say, with maybe navigation and stuff.
MattGPT says:  but you find you know you. You'll find that
MattGPT says: the first one. Here's the first part passenger list. and it's given you already in a data framework thing that's missing here. Is that
MattGPT says: sometimes, you know, it won't work that well, but in this case they've used visuals.
MattGPT says: They use the visual code. Yeah, to indicate. Yeah, one means they died.
MattGPT says: Food means that they survived.
MattGPT says: So maybe the liveboat actually gives you that it doesn't always work that way. So these better are relatively clean, and and I think the HTML
MattGPT says: 3 days and hour work. but it could be that you need to get into that. But then this would be in a case of where you would start to look at the structure here
MattGPT says: identify. If there's a table
MattGPT says: that's cool, windy table.
MattGPT says: all of these things are searchable. Okay?
MattGPT says: so
MattGPT says: you can follow those same, those same sort of steps which would be.
MattGPT says: Do a get request.
MattGPT says: The passengers of the Titanic. Would you pay  And then.
MattGPT says: yeah, the gmail
MattGPT says: turn it into
MattGPT says: a beautiful, superb Jack.
MattGPT says: And then what we're looking for
MattGPT says: is all of its table elements
MattGPT says: have. And then you give it a dictionary. Here we have above attribute where the values are wiki table and forward.
MattGPT says: That actually does a really nice job of finding just the first, second, and third class tables. Right? So then, once you do that, think about well, how would we do that?
MattGPT says: We go back and look at what the table looks like?
MattGPT says:  yeah. you can see that this we found we found a table. and then the structure within it is that we've got these.
MattGPT says:  we've got these Pr elements, which is each row is represented as a.
MattGPT says: He asked.
MattGPT says: Okay? And on that Pr element the ones that have the blue background have a style attribute, and then a background color
MattGPT says: that gives us certain codes and style the data that person survived. Or
MattGPT says: so we're just saying, Okay, we're gonna pull that out yet.
MattGPT says: We want to hear road by road
MattGPT says: and get the data from each of these columns.
MattGPT says: and then, if the particular row has a blue background.
MattGPT says: then we're going to add an extra attitude.
MattGPT says: Because that's the scraping sort of
MattGPT says: approach to getting the same data.
MattGPT says: Okay? So I'm happy to kind of ask any past any questions on that, or because after that, we're already use the rest of the time, for is
MattGPT says: in your group. A talk to you got to fill out the
MattGPT says: the structure and the boat is okay. So if you have, if you're still trying to decide on your project, I'm happy to come and talk to you about your ideas. But if you kind of settled on that, and you know, the first step is actually to go and get your raw data
MattGPT says: so hopefully the Cfv. Or go out and eat if you have a scraping.
MattGPT says: if you know that you're going to need to do some scraping
MattGPT says: or extraction from Pdf, we talked about that on Tuesday.
MattGPT says: Okay, I can come talk to you about that. So we can get started with that.
MattGPT says: Okay. But if you found data sources that are on data.gov, then just download them in Psv and then upload them to your data folder
MattGPT says: to start. The. That's what I want to start working on now is
MattGPT says: organizing the data.
MattGPT says: Thank you. Okay,
MattGPT says: United States
MattGPT says: with innovation. So your businesses?
MattGPT says: Oh, I see
MattGPT says: future design. And who else is on to me?
MattGPT says: Maybe I'm wrong. But like.
MattGPT says: yeah.
MattGPT says: it's like, that's actually what? Yeah.
MattGPT says: it depends on the function.
MattGPT says:  oh, really, number one, you're
MattGPT says: yeah.
MattGPT says: right?
MattGPT says: Yeah. I remember. So how was that built in?
MattGPT says: I believe. Yeah, I believe
MattGPT says: the 6 tables like 3
MattGPT says: and also
MattGPT says: And
MattGPT says: okay, so you got the data from the
MattGPT says: so it was
MattGPT says: so yeah.
MattGPT says: perfect.
MattGPT says: Okay, so yeah, pull.
MattGPT says: yeah. But then, like, you can try our data or
MattGPT says: out of listening
MattGPT says: yeah.
MattGPT says: if willing to find a mom, see whatever
MattGPT says: I'm sorry.
MattGPT says: And you're going to.
MattGPT says: Yeah, you're right.
MattGPT says: just wondering where? Exactly.
MattGPT says: So inside an Alli element, and
MattGPT says: and then
MattGPT says: beginning
MattGPT says: that. And then you would find.
MattGPT says: I mean, we justability. Yeah. Oh, my God.
MattGPT says: yeah. So we put in data analysis, right? Even though.
MattGPT says: yeah. So I think that
MattGPT says: it's like no response to a R asp, and I guess it's
MattGPT says: request we're gonna have. Yeah.
MattGPT says: yeah.
MattGPT says: you should.
MattGPT says: Okay. So then that would be the webinar and be also. I guess, 200 grapes that do like
MattGPT says: below 20.
MattGPT says: And then we're gonna do like.
MattGPT says: So it's gonna be forced.
MattGPT says: And then beautiful students, we have a pose sales.
MattGPT says: Alright. And then
MattGPT says: okay. So that's how it will look back in
MattGPT says: okay, cool.
MattGPT says: Wendy
MattGPT says: versus.
MattGPT says: That's like, those are 2 people. another scrabble. whatever. Olympics. Yeah, I think you understand.
MattGPT says: I didn't think so.
MattGPT says: Just put it in, yeah. But you got it. Okay, yeah. So then, but you got it for that. And let's go back to the website. You had a way of just
MattGPT says: right? Yeah, yeah.
MattGPT says: right? So it was this. URL,
MattGPT says: yeah, yeah, yeah. probably.
MattGPT says: Okay, okay, so that will be
MattGPT says: to look at like altered by school catchment. And so filtered by the League. Could I be altered by the in your cycle like the H rout?
MattGPT says: Well, so what you want to do is like you wanna say and specific questions.
MattGPT says: okay? But then,
MattGPT says: and that, like our their special edition, we were making a lot of speech earlier.
MattGPT says: Yeah. Okay?
MattGPT says: Right.
MattGPT says: high school drops, more expenses. Oh, okay, public schools.
MattGPT says: Oh, they released.
MattGPT says:  okay, so that gives me
MattGPT says: unfortunately diverse. I know we have a victim.
MattGPT says: But yeah.
MattGPT says: equal. And
MattGPT says: I think if we do, we have like assignment.
MattGPT says: Yes.
MattGPT says: pleasure.
MattGPT says: not useful.
MattGPT says: Okay.
MattGPT says: right? That's what we are looking for. Basically.
MattGPT says: yeah, yeah.
MattGPT says: a
MattGPT says: that works that works.
MattGPT says: Okay? So let's say this like,
MattGPT says: oh, okay.
MattGPT says:  we'll return them.
MattGPT says: No.
MattGPT says: okay.
MattGPT says: nicely.
MattGPT says: So if you would take one of these.
MattGPT says: yeah.
MattGPT says: here's the other stuff.
MattGPT says: That's huge.
MattGPT says: Okay? And then
MattGPT says: there we go. We just need to update here. It's like.
MattGPT says: who are loop through all the games of the town here. And like, I said.
MattGPT says: Where sorry.
MattGPT says: what's the meaning of that?
MattGPT says: Yeah. Okay. Which ones?
MattGPT says: Sorry. But one directory?
MattGPT says: You like the state like
MattGPT says: that. That will be impacted by
MattGPT says: we can make
MattGPT says: yeah comfortable size, you know. Like.
MattGPT says:  you?
MattGPT says: Oh, I have some good
MattGPT says: one, professor it?
MattGPT says: Yeah, we're right. I don't know.
MattGPT says: So here you go.
MattGPT says: only one society
MattGPT says: incidental, Stephen
MattGPT says: on it, listening to one by one.
MattGPT says: Oh, right now we're like
MattGPT says: so getting this page. But
MattGPT says: it's not like, it's not getting the more.
MattGPT says: That's not. That's not quite difficult. Yeah. But you need to know and like what you want to get out of it. Right? Yeah, like, this is a long blog, and they're gonna be very
MattGPT says: provide. You want to have a row which represents a game.
MattGPT says: What do you want terms. let me find they're there could be like
MattGPT says: gang
MattGPT says: idol. Turbulent. Id
MattGPT says: alright. So then each row represents
MattGPT says: turn in a game by a player right? Or it could be that what you're trying to do is a aggregate level, which is each game. Get the same row, and somehow you have columns to represent.
MattGPT says: who wins and say, who loses.
MattGPT says: maybe
MattGPT says: analyze the log and figure out what it is that you want to say about this game
MattGPT says: in a table. Brother, right? Okay.
MattGPT says: yeah. I got. I understand what's going on when you came over.
MattGPT says: That's what you want to think about for hanging on.
MattGPT says: Okay. So I guess that works.
MattGPT says: whether we have
MattGPT says: that is provided.
MattGPT says: So many other metrics will city, even though we pick like 5
MattGPT says:  Oh.
MattGPT says: cause you know the
MattGPT says: I mean, not.
MattGPT says: But do you think it's not?
MattGPT says: Yeah, that's true.
MattGPT says: See if there's yeah usual, like a national like. why, you can data set up. So what is what is like.
MattGPT says: I know.
MattGPT says: yeah, I think this is
MattGPT says: funds. Cause like, we're we're in the city. It's kind of like a
MattGPT says: trying to identify areas which are better President or Monday or
MattGPT says: right? And so then say, looking at Philadelphia, there'll be some that's awesome.
MattGPT says: Have more areas you have.
MattGPT says: And like possible, Philadelphia, maybe don't have as many right? So it's kind of like down at that level. But like the State, yeah, State level is tricky, because that's too big. I think so. Then
MattGPT says: I guess that you could do
MattGPT says: what the comparison would be like. Some sort of angle saying is.
MattGPT says: they try to rank the cities right or
MattGPT says:  I think we were trying to see if it was like statistically significant in all 5 cities.
MattGPT says: And
MattGPT says: yeah, but then
MattGPT says: should only exist. So yeah.
MattGPT says: so maybe start from the other direction, which is like, you're you're kind of like looking at an indicator or an independent, variable right with this thing.
MattGPT says: But like, what are you? Dependent variables? Right? So is it, child obesity, is it, child? You know it could be anything right?
MattGPT says: Be at what level that you can get? Can you get those? And then you'll see that find out the same? Right? You
MattGPT says: not 50 states. Yeah, you know, at least point.
MattGPT says: But the travel state level is that then you've got like urban suburban.
MattGPT says: Right? So then it'd like
MattGPT says: that helps up. Usually you'll find the counties that are reasonable.
MattGPT says: like middle ground. Okay, right
MattGPT says: under that 3,000 counties. So you've got a good number of rows right? And data points.
MattGPT says:  and then should be a lot of county level metrics. Fine. Okay, look at that. Right?
MattGPT says: Pretty like.
MattGPT says: wanna if you wanna even have a political variable, because you can go back like last election or previous elections like, does it vote red or blue right? And then should be able to find metrics at the county level for investment in various things. Right?
MattGPT says: I've seen a data sheet before to like proportion of the population at the county level that have health insurance.
MattGPT says: those sort of things. Alright. That sounds interesting, right? So I mean, that would be the way to go. And it doesn't have to. You know, it could be any metrics and you're interested in. But, like you want to do a help related one in this country, I would suggest, like
MattGPT says: alright. Check it out.
MattGPT says: Identify some potential dependent variables, and probably a good idea to have more than one right if you find them, you know. So it could be obesity, it would be drug. Use the other sort of thing and then start to look for like what the independent variables are
MattGPT says: have. Yeah, sure. What's that?
MattGPT says: Okay, yeah, sweet. Thank you. Thank you so much.
MattGPT says: But yeah, I think so. I mean those those
MattGPT says: in in the past. This was project to work out. Well, yeah, you want you want to.
MattGPT says: You want to have more than 5 points for sure. And you want to have more than maybe more than 50.
MattGPT says: Okay, you could still then have a like. We look at this at the national level. And then our data story goes down to deep dive in Philadelphia or East side of Chicago. I think that's what we should do. Yeah.
MattGPT says: alright.
MattGPT says: basically.
MattGPT says: Where?
MattGPT says: I want to flip the code
MattGPT says: on this. Okay? So you call on that your cmat. You just put up.
MattGPT says: What have you got? Okay? But but underscore R, at the end of that strength
MattGPT says: reds under 4 R, and the way they're met from it.
MattGPT says: That's it. Okay? So so the column that
MattGPT says: so I think vines are with oranges. But that's fine. You want to use it as.
MattGPT says: yeah, and then go out.
MattGPT says: Yeah. So then the hour. Yeah, the other thing you want to do that get rid of the yeah, standardize. Standardizing be meant. So female be mapped right into that. So I think we did like 30 to 85. Okay, yeah, you go
MattGPT says: like hard coding it. Basically. Yeah, I agree handed. But if you wanted to do like the whole well, they DM, find them in, find the maximum, or, however, the 2 years that you're showing.
MattGPT says: you'd have to find the men and the map out of that which? Yeah.
MattGPT says: okay, that's great.
MattGPT says: is this on one. Or should I just do show? And then dashio no, if you
MattGPT says: no, you can just do plot. And then what? Okay? Awesome. So then we're being getting this. Yeah into this
MattGPT says: cool.
MattGPT says: cool. Okay?
MattGPT says: Alright, good. I'm really sorry. And yeah,
MattGPT says: so yeah, I looked, I had a look over that package survivors up.
MattGPT says: It looks like it's mainly them checking the data together. There is also some analysis like dashboard and stuff
MattGPT says: Saxon to be like Saxon City like Edie about it. So but I wanna get a little bit more than Hey, this is, gonna be a cool project, and like, what are some of the questions, or what are some of the things? Yeah.
MattGPT says: and so it's a really nice kind of like media com project, because it's like.
MattGPT says: so how you edit things. Yeah,
MattGPT says: But yeah, I'd like to like to see a little bit more development of the idea. There's yeah, there's a ton of data right there, right?
MattGPT says: I think they're also definitely a ton of questions to be asked since i'm not
MattGPT says: like
MattGPT says: or as nerdy of an audience, that that show can attract
MattGPT says:  somehow has seen very little like good analysis. Right
MattGPT says: as a noticing right
MattGPT says: I don't know, if you ever end up on like today.
MattGPT says: like, I've never watched very fair. But I mean, like on when you end up on like Reddit and like
MattGPT says: on the for a online where people are like, look I did this little visualization. It's like, but you did.
MattGPT says: There's so many problems with your
MattGPT says: with your approach to data science and this that I'm like. I don't know if I can call any of your research legitimate. And then you see all the comments on, and people are like extrapolating huge findings from it. And it's like.
MattGPT says: yeah, so I'm serious research on stuff. Yeah. So I mean, I guess it's a difference. You've got an update that But yeah, like them. Yeah, big likes.
MattGPT says: I want to see you go through all the steps the first thing, and then sort of exploring the data, doing the Eda on all of those things
MattGPT says: and say, like, what are the potential? Choose one or 2 other people get. I don't know. Like, if in the data, can you create a social networks like honest season of survivor
MattGPT says: in in, in the data? Is there a way to show that these people sure interact? Right? No, and I think but that probably is right, which is like, you know.
MattGPT says: contestants 1, 2, and 3.
MattGPT says: That sort of thing. I don't know if the data. Does that right? I'd have to check the actual. Yeah.
MattGPT says: because you, you might be able to make a. you can make a network like that. Then, yeah. do you have
MattGPT says: access to our data in here? So I'm not taking up my laptop cause I think if I remember all of the files that it has access to. I think those should be in our group's data holder at this point.
MattGPT says: If you want to move, Group 5,
MattGPT says: okay, this is kind of stuff that we got right.
MattGPT says: Boot mapping. No, I wonder.
MattGPT says: So this is the first step right? We can get, you know, if you can get these out. Turn these into ditch to to
MattGPT says: Csv, right? The most safe. Yeah. On the the mapping idea, though, if you go all the way down both history could be one way to approach mapping right? I wonder? Yeah, they have.
MattGPT says: I don't know if it's in here, or if I like, I think it's on their online page. They get they do have like good descriptions of what each
MattGPT says: like data set is, which is nice.