User: ---
File: /Users/erayack/Desktop/output_audio.txt

A couple of students asked me for suggestions on how to get better at data modeling.
Data modeling is a big topic. It's also very challenging to teach. It's one of those things where we have less like established understanding.
Definitely the more you try and become scientific or prescriptive about it. The further you can view off track.
Definitely if someone is trying to sell you a book, a course, a methodology, you should be very skeptical.
That's not to say there isn't stuff to learn from people who have tried to think about this clearly.
Absolutely there is.
But you can just be really prone to going too far down the rabbit hole because basically everyone's trying to sell you on the idea that there's a simple fix to your data modeling problems.
There's absolutely not.
So I'm going to give you some suggestions based on what I see are common problems, particularly with my students who tend to be people who've done some web development before and the orientation for that a lot of the time is around the object oriented languages and all realms a lot of the time and relational databases.
And look, they can actually set you up for some good intuitions, but also some bad intuitions.
And so I wanted to give you some basic suggestions that may enable you to take a couple of steps forward.
Maybe start to think about some different ideas or different styles of modeling data different ways of thinking about your business problems.
Maybe some search terms that you can pull the thread on. But again, I want to emphasize you probably do not want to end up in a place where there's a conference where everyone is trying to
do a particular style of modeling where you're using tools for a specific style of structure in your data or whatever.
Hopefully you have my hopefully you've heated my whole money.
So all that said, what do I see as the two most common issues?
One is that I see data models where really implementations ultimately because when I see them as an issue that they're there as postgres tables or whatever, where it's not clear where the source of truth lies, where there's a kind of modeling.
And it'd be possible to derive something from two different locations and the system is not clear and so therefore the programs are not clear about what is actually true.
So if you imagine a situation where let's say you've got an unread message count and that's a field on a table as an integer, you know, 15 unread messages in this context.
But also it would be possible to compute the number of unread messages by looking at the messages that have arrived, their timestamps and one was the last time that someone did a read of that channel or whatever.
Then you've got this this mudding of the waters. There's no clear source of truth and that's not to say you can't have these two pieces of information in your data model or in your system.
But if you're not very clear about which one is derived from what then that can cause bugs.
You probably can't have these kinds of bugs before where you just have this kind of disparity because part of the system, some of the programs are looking at one and some are looking at another.
So that's one major thing. Another major issue is just data loss.
Maybe not like catastrophic data loss, but data loss in the context that you want to be able to do something and you don't have the historical data to do that because you're mutating information in place.
And so you would just never never model the updates.
You model the kind of state as it was at some point in time and you model mutations updates to that the you of crud.
And so you were just changing this field all the time to say the unread message count.
And maybe you are incorrectly computing something and couldn't redrive the result.
Or you just want to have some information that that's existed in the past but wasn't part of your model as an event as a kind of transition in the state.
And so you can't recover that.
Interestingly, these are both kind of the same fundamental issue and so that's when I want to speak of.
So let's say as an example and you can really really any example that let's say you have a you try to model the system with.
So I was going to say like a students and and classrooms and a classic example but because you are students in a classroom maybe.
For just a tiny bit of right let's think about employees in an office let's say and your part of a team employees are part of team so you might think you intuition might be to say hey well obviously there's like an employee.
And so you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know,
you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know
you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know
you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know
you're thinking about this in terms of entities that have fields and a kind of relationship which is not considered a first class system. So, uh, this is the default way of modeling and my like one weird trick.
I want an idea that I want to instill to you in this short video, which again, we could spend a lot of time talking about that in a moment, but in this context of just like a brief advice. Um, this, this is already leaning towards some issues.
One major issue here is that we've not considered this to be a first class system and in a lot of our arms, you don't even need to do to consider this as a first class system.
You just, you just say that there's a many to many in relationship between your employee class and your team class and the underlying table if this is a relational database is constructed for you, uh, under the hood and you know, even thinking about it.
The issue with that is that the acts of an employee joining a team is a thing that happens. And if you're not modeling that, like the, uh, you know, you may want to think of this as like an employee team join or something.
Or like, um, uh, you know, one of the naming things as hard. I could do a separate video just on that. But, um, let's see team.
Maybe new team member or something new team member. So this is the act of an employee joining a team.
If you want to think of this as an employee joins a team at a date.
You're creating a new one of these that each employee joining a team. You naturally have the fact from here that employee could join multiple teams because you have one new one new one of these each time.
If an employee leaves a team now you're prompted to think, well, how do I model this? Do I delete the new team member?
Well, no, because this is like an event. I'm thinking of this as a thing that happens. Like this is a noun as well.
It's just not like a, a basic noun like a toddler's thing of an employee or a team. It's an event.
But events is a thing that happens as well. It's something that did happen. We don't want to lose the fact of that event happen. And so we think of this as, um, like having a team member departure as a separate entity, a different event.
So whenever a new team member joins, let's create a new team member, um, a theme record of role in the database, whatever a corresponds to in your system.
But this is a thing and a point to an employee and a point to a team, but we have the data which it happened. And this whole thing is one immutable event, I think that happened.
And we can accumulate more of these. And then if a team member leaves a team, then we can make one of these with a date and so on.
And at any point in time, if you want to compute the thing that you, you maybe wanted in the first place, we're focused on in the first place.
The current membership of a team. You can do that by looking at what are all the new team members ship, and subtracting or like,
removing from that set, the departures for that team. Right? So it's computable. Now you're going to tell me that feels like hard work.
Now I need to write a SQL query. Does my orom support this? To me, these are the wrong questions to ask.
To me, the right questions to ask are, am I going to be able to clearly determine everything that I need to?
Am I going to lose data? Yeah. Am I going to be able to figure out historical team membership? If I, if there's an issue,
and there's a spiritually created team member departure or something, have I, like, accidentally overwritten? Have I accidentally deleted something that was meaningful information about the past?
Or is that still there? And I'm just going to compute it differently.
So to come and say, well, it's hard to do that. Look, I got to tell you, maybe the positive way of thinking about this is, I believe in you. You absolutely can.
It's not that hard. It like it takes a little bit more work, but once you figure out how to do that kind of thing,
it feels straightforward again, and it becomes as easy for you as the worst data model feels for other people.
So it's just like a threshold that you have to pass. The, like, more negative way of talking about it is we are professionals.
If we are in a situation where it just requires a little bit more work for us or a little bit more competency for us to deliver a better thing to the user, it is our job to do that.
That's our job. Like, if you engage in architect and they say, well, it feels hard. Actually, this just happened to me last week.
It feels hard to use 3D modeling software. I want to just use the 2D modeling software. You're going to say that the architect, look, I'm the customer here.
Maybe I'll go with the different architect. I want you to be able to do the thing that's better for me. And I'm sure once you use to the 3D modeling software, you'll be able to do it.
You'll be able to do it if everyone else. And it'll be just as easy for you.
Rands over, all that's to say, yes, it requires a little bit more work. It's not going to be on a kind of normal scale system computationally challenging.
It's not going to take too much compute to much storage or whatever resource you're worried about, it's going to be fast.
It's going to be fast enough. Particularly once you understand your database well, and if it gets really big, you're adding joins and you're adding indexes and stuff as a way of talking about it.
And it's going to be a little bit more efficient. And then you're going to do that. And you end up with a better model.
And the result is by thinking of the event in the system, we have a clear way to determine the answer to our question.
And also, we have this historical data from which we can re-compute the answer as needed.
But this isn't just about many to many relationships. It's about mutation. It's about thinking about the events that happen as first class citizens.
Writing them in an append-only matter and then in an append-only manner and then deriving our kind of current state from that.
And another example of that here is this email field.
This is a very natural thing to do when you're writing a crowd application and you're thinking of updating when you're working in an RRM.
Because it's normal. It's sensible to mutate memory in place. We're not trying to create things in memory that are entirely appended only in a mutable and so on.
Or there are some languages and ecosystems that push you in that direction.
It's natural for us. It's a right-acclass that has a field that's mutable. And natural for us to write an employee class with an email field.
But if you're using an RRM and that is translated into a table where the email field is updated, that may be problematic.
Now I'm not going to say it's always problematic, but consider a situation where the email address changes.
You are updating in place. You're losing the old email address. Is that okay? Maybe it is. But if you haven't thought about it, well, maybe it's not okay.
Maybe there's going to be some point in time where you need to look at the historical email addresses. Imagine scenario. And I see this kind of thing pretty frequently where an application that's just been written is fine with a user that has an immutable email address.
But then it gets to the point where they have like 40 shares or something. They have people making fake accounts.
And as they're changing the email address and committing their fraud, you cannot associate the previous email addresses from other artifacts where the email address is present with your transactional data in the database.
Now maybe what people tend to do in that situation is to keep this main model, this transaction model, immutable and say, hey, whenever someone changes their email address, we're going to just keep doing what we're doing, but we will log the fact that the email address changed.
Maybe in some situations that's going to be fine. Maybe you don't want the changes in your transactional database and you're confident that you'll be able to search the logs for whatever for your audit trail.
And I'm not saying this can't work, but a lot of the time in those situations people forget to do the logging people are not so disciplined about the logging because they're anything you may have that order trail thing later.
Or just more broadly, they're not thinking about the consequences of the mutation and when they would have to do the logging and what data they would want to retain.
So if you rearing yourself to think about this as maybe in your data modeling, you're thinking of this as not a user that has an email, but that we have a user entity, a user is created, let's say, and we have like an email address claim or something.
It's like the events that happen is the user told us that they have an email address, and then an email address confirmation, let's say.
But we thought these things are immutable events. A user at this point in time told us that they wanted an email address, that they believed that they have this email address.
Okay, so we sent them an email confirmation challenge. They clicked it and it worked. Now they have a confirmed email address.
Okay, these are the events of the system and from that you can derive the fact that a user has this email address as far as our system is concerned because they went through these events.
There's no opportunity for data loss because each time one of these events happens, we just add a new route. We're not updating anything in place.
Again, maybe you tell me this is hard, this is messy. I'm not familiar with this kind of thing.
Okay, I believe in you, you can step up to the challenge. I'm also not saying you necessarily need to do it this way, but if you're instincts in data modeling, I'd say, what are all of the events in my business that I care about.
And I will write them as excuse me, first class citizens and then treat those as my entities and then derive the information that I need, like the users current email address from that.
That's going to be a better first instinct in terms of at least avoiding these major problems that I see in data modeling.
I am also fine with you saying, hey, I want an employee field with a mutable email address, but I want you to have a story around, well, what happens if we want the historical data, where do we get that from?
If that's a separate table and now transactional database, are we going to be very clear about what is derived from what?
I would prefer you to have this as your default model, the events, and then for you to very explicitly say, hey, the email field on the employee is something that we don't want to have to recompute each time.
And so we have this as a very clearly stated, maybe in documentation source of truth, and then we have a trigger, which updates based on very explicit rules that are in like a stored procedure or whatever in your data basis, and there's going to be something like this.

And so, the latency that this email address that's a snapshot of the state is correctly derived from a source of truth, which is immutable.
So all of that's to say, you don't need to go hardcore now in this part of like, let's only ever have immutable data and always compute our state from that all the time, just in time as part of the database query.
And tell you that you are going to get away with that for a long time, for very large tables, kind of you might surprise yourself how large the tables can be, and the main barrier to this is an immediate lack of familiarity with the kind of tools that you need to be able to do this and on the number systems themselves, they people are doing quite a lot.
Even just to give you a kind of intuition around that, like the considered the example that I gave earlier of the unread message count.
A lot of people would model this as we have like a room entity, let's say, and a user entity, and a new entity type that's like user unread message count.
Maybe you split that out, you used to be a user has many rooms or something, and then over time you think of that more and more as a first class entity, like a membership of a room.
And then eventually you think that the user room unread message count should be a thing.
But if you think of that as an integer and update that integer in place is highly likely that you get this wrong, and it's highly likely that you have the information elsewhere that it could be computed from.
And so to normalize to have it in a normal form, you would want to compute the unread message count from all of the messages.
You might think what happens when there are a lot of messages, you're absolutely right.
As you get to have a lot of messages, it will become computationally challenging to compute the unread message count per user from the messages.
But what order of magnitude do you think that happens?
Is it like 10,000 messages in the system 100,000, a million, 10 million?
I will tell you that it's a, I mean, obviously there's a lot of other context, but just to give you a bullpock,
on a system that I worked on, then had this, we didn't normalize that 100 million messages across the system.
So until we got to 100 million, we, we were thoughtful about indexes.
Well, actually that was it. We were hoping that indexes.
And we, you know, we tolerated this particular query for the unread message count getting into the tens of milliseconds.
The, you know, maybe in your context, maybe there's too much tolerance, but in terms of our answer ends for that particular computation.
That was, that was okay.
So once we found that intolerable and wanted to fix it, we didn't normalize it.
But we didn't normalize using triggers, we had very clear rules.
We were clear about what the source of truth was.
If the unread message count ever a zero bug or something was computed incorrectly, we could rerun the computation from the source of truth.
All of that stuff was in our transactional database anyway, because we wanted those old messages.
We didn't want to move them into cold storage, like we wanted to be able to use that for other calculations as well.
And so that stuff was going to be there anyway.
And to have a very clear system where we could re-compute that with no data loss, always just adding records.
That, that was advantageous to us.
So we really didn't need to solve the denormalization problem until about 100 million row kind of level.
So it's not, you can't argue from the beginning that this is due the inefficiency.
There are, there are means that you have for working around that, these systems are capable of what more than you would, you necessarily imagine.
And in any case, having that first orientation of like, what are the events in our business that we care about?
Like, even if that's all that I convey to you, when you started your data modelling, don't think about what are the things in the way that a toddler does, there's a ball and dog or whatever a dog has a mouth.
Think about the events, what happens, then you are nudge towards a system that is more careful about the source of truth and less likely to lose your story.
Exactly to lose historical data, which I see is the major problems.
Again, you could, there are search terms here, like you could look into events sourcing into CQRS and so on.
I really do not want you to go, just wholesale down these paths because if you just take this idea and don't compromise on it, it is quite challenging.
You end up having a lot of events and a lot of recomputation.
And basically, if you're dogmatic about what are suggesting that you do, then you may be unnecessarily making your job way harder.
But you may want to look at these communities and derive some, there is something to learn there, at least as a kind of counterbalance to the default community of object oriented programming, our imitation in place, crowd applications and so on.
As a kind of meta-less of there, as well, it's maybe don't isolate yourself within one of these communities because there are other ways of thinking about how you model your business, how you model your data, that may be a better compromise in any given situation.
But, explore these ideas, maybe try and have an inclination towards a modeling events rather than things, have the current state, the computation, derived from the things that happen and you'll probably end up in a space that's a really better compromise overall.
And then once you do that, maybe you want to try going hardcore in this direction and using a system that's designed for events or something or whatever, and then backtrack from that as well, at least going through that process, you will then have a kind of appreciation of where you want to make the compromise.
It's like, I would maybe if I'm prototyping something, just have email as a field and employee and it's mutable and it's no big deal, but I do have a voice in my head saying, how important is the historical email?
How important is it that I remember the whole thing? Do I log that somewhere? Do I put a note saying this should actually be an email created or email requested record or something?
Or maybe I become more and more comfortable with that kind of drive data approach and I did that from scratch on a system that I think is reasonably likely to move beyond a prototype and I want to just get it right from the beginning.
At least you can choose where on the spectrum to see, but that's more or less it might.
Again, the caveat is that for my students who come from already having a reasonable inclination of how they should model things, but having a little bit too much of a culture of mutation, the main advice that I give them.
I could say you need to normalize your data more and this is a very common advice. There's no clear source of truth.
But then if you pull the thread too much on that, you end up in all the normal forms and all this theory that doesn't really help people in practice come up with better data models.
You may want to explore that, those are search terms as well of normalization, but the core idea there is, do you have one single clear way to identify what is truth of what happened in your system?
Is there one of them and precisely one? If there's zero, it's like, will you have lost historical data? If there's more than one way to drive that data, then that's a recipe for disaster.

Assistant: [system: file uploaded]
User: As an expert in the subject matter, please provide a comprehensive and structured set of course notes based on the following transcript. 

Please ensure that all critical points are included, and present the information in a way that facilitates understanding and retention.

Assistant: