Chiba University Leading Research Promotion Program
Online International Seminar (in English)
Positive Political Psychology International Seminar 3:
Social Fairness and Wellbeing
Measuring Wellbeing through Social Media Language

Date & Time：24 Mar. 2023(Fri), Japan
Abstract: This is a research essay based on the transcript of the third meeting of seminar serials on “Positive
Political Psychology International Seminar: Social Fairness and Wellbeing”. Professor Lyle Ungar(Professor of Computer and Social Science, University of Pennsylvania) who is well versed in computerscience and whose cutting-edge research combines scientific and technological directions with positive
psychology, addressed how to use social media language from people to better understand what wellbeing isand how to measure it during his presentation entitled “Measuring Wellbeing through Social Media
Language”. Then discussion was initiated, and research ideas were exchanged afterwards.

Moderator

・Prof. Hikari Ishido (Chiba University)
Panelists
・Prof. Lyle Ungar (University of Pennsylvania)
・Prof. Masaya Kobayashi (Chiba University)
・Assoc. Prof. Xiang Li (Chiba University)
・Dr. Xiaofang Zhang (Chiba University)
・Mr. Alfonso Torrero (Universidad Autónoma Metropolitana)

Organizer: Chiba University Global Fair Society Program

Page 1 of 22

Introduction (by Prof. Hikari Ishido)This online seminar is entitled “Positive Political Psychology International Seminar 3: Social
Fairness and Wellbeing”. This is our third meeting for the “Positive Political Psychology
International Seminar” series focusing on social fairness and wellbeing. The focus of ourresearch group, Chiba Studies on Global Fair Society is centering on challenging andovercoming unfair practices on all scales, investigating issues such as gender inequality,
widening income gaps, migration, and the collapse of reginal and marginal communities, bothin Japan and around the world. In doing so, we seek to identify the development, transformation,and limitations of existing welfare state models and provide empirical evidence for how a fairsociety can be achieved in the 21st century. Our special emphasis will be placed on “wellbeing”
and values because fairness is intrinsically related to people's subjective judgement. Objective
equality does not always guarantee a subjective feeling of fairness. The vision of a fair society
should include both objective fairness and subjective sense of fairness. We are planning tobroaden our networks of academics across the globe.
Today we will have Professor Lyle Ungar (Professor of Computer and Social Science,
University of Pennsylvania) to talk about how to use social media language to better understandand measure wellbeing. Then Professor Masaya Kobayashi will initiate the discussion so that
other participants could join in and exchange research ideas afterwards.
Measuring Wellbeing through Social Media Language (Presented by Professor Lyle
Ungar)
What I want to talk about is how we can use language to better measure and understand wellbeing, which I think is a big topic and very important. I'm going to talk very much aboutmeasuring well-being in English and America today, but we're starting to do the same thing in
China, and Japan, and other countries. And one of the reasons is that now I think the techniquesthat we've slowly developed in English in America can be applied across the world. I think it
helps to both major well-being and understand it. And one of the questions I won't talk about,but I'm most interested in is how do the contributing factors to well-being differ around theworld? I think cultures are not all the same. And we need to understand how people differ. My
goal is simple but difficult to use language from people to better understand what well-being isand how it varies across different individual people and across different cultures.
We look at language at lots of levels. We usually look at single messages, one text message,
one tweet, one Facebook post, to see if people are happy or sad or excited. We also look atconversations which I won't talk about today. We will look at text messages. How do people
talk differently to their husband, or wife, or to a stranger? How do depressed people talkdifferently? And I will talk about traits that people have, to see if they're extroverted or
introverted, neurotic or well adjusted, depressed or not depressed. And we can measure it inmuch lower larger level and having access to billions of tweets. There are variations across the
US or across the world. And how do people show their well-being different? How does itchange over time? We watch during COVID, as well-being went way down, and is now happily
coming up again. People are happier now that COVID is gone, not surprisingly, but we can get
more detailed measurement.
As a psychologist and a computer scientist, I'm going to talk about tools, and language as a tool
for measuring things. Over time, people start with little telescopes and build bigger ones andbigger ones. In some sense, I want to build telescopes and measurement tools to understandpeople through their language which I think is a great way to understand people. When you talkto someone, when you send a text message, you reveal something about what you're feelingand what you care about. People rarely say that they are feeling so depressed, or they really

Page 2 of 22

don't like someone. People don’ say everything, but they still reveal something about
themselves. Then we can use language to measure people.
In psychology, and in political science, people have asked questions for hundred years. They
take surveys by asking “how do you feel about this candidate? What do you care about? Who
do you want to vote for? What's important to you?”. But it's expensive and slow which is also
hard to do every month. Increasingly, we have large computers which have measurements fromTwitter, Facebook, Instagram, Reddit, TikTok, all these platforms that people arecommunicating on. In some sense, these are alternates to surveys but not the same as they allow
us to give a different measurement. If I have your Facebook and your Twitter feed, and you'reusing them while some people are not, I can know more about you. And in some sense people
will sit around the campfire and talk to each other, and in the everyday conversations you havelike this, you reveal what's important to you, what you are talking about, what you care about,what you are worried about and what you are excited about. I would like to show how we canmeasure well-being from people, and how we can maybe even understand well-being
differently through language with several examples. And it will set the stage for future work
for asking questions of how well-being differs across countries.
What I do mostly is to ask tens of thousands of people to take a survey to measure theirpersonality, character strength, depression, and happiness, seeing their language from Facebook
or Twitter. I would like to show some studies correlating the language people use and theanswers they give to questions to try and understand how people talk differently who havedifferent personality or well-being. By counting words out of Facebook statuses, lots ofnumbers will show how often each word shows up. At the same time, survey responses also
show the numbers. Then we correlate every word so that we can understand something about
how the languages language people use predicts well-being.
For example, this is Americans talking on Facebook with the words they used, correlated withhow extroverted they are, standard personality, one of five factors of personality. The bigger
the word, the higher the correlation with extraversion (Figure 1). And the redder it is, the more
frequent it is. And the more blue or grey, the less frequent. What is extraversion? Extroverts
are people that like spending time with other people, they like going to parties, they like otherpeople being with other people, which are gender-controlled and age-controlled. They tend touse words like “amazing” and “cant wait” and “love you”. And lots of slangs that are not quite
standard American get in instead of getting a more casual or informal piece such as “lovin”,“doin”, “Sooo” or “Soooo”. This is a style of talking of people who want to go out and have
fun with lots of people. It’s funny because these are people who “cant wait”. And in normal
English, it's “can’t”. But these people are being so excited, and they're typing their messages
on Facebook dropping the apostrophe, which might make sense.
Extraversion

Intraversion

Figure 1

Figure 2
Page 3 of 22

What's the opposite of extraversion? Something and one is not better or worse but a different
personality. The opposite is introversion (Figure 2). What do American introverts talk about?
Interesting, we can see lots of Japanese culture such as “manga”, “anime”, even the word
“Japanese”. It is that Japanese culture is very popular among introverted Americans. “Drawing”,“reading”, “books”, these are introvert activities that you could do by yourself, working on
computer and the internet. Introverts also tend to use hedge words, for example, “apparently”.
They tend to say, “I'm apparently doing very well”, rather than “I'm positive, I'm doing verywell”, which is what extroverted to be sure of yourself. It shows that introverts like reading
Wikipedia. But unfortunately, introverts are a little more inclined to be depressed than
extroverts. So mostly, introversion is good academics who tend to do science while
extraversions are likely to be salespeople who must deal with people for a living. It might makesense that if you're scholarly, you need to be a little introverted because you need to enjoy timeon your computer and reading books, and if all you want to do is spend time with people, you're
not going to be a good researcher. We could notice that we see something about the life of
introverts and extroverts from the language they use.
Here is another example to show how word use captures personality (Figure 3). What doneurotic Americans talk like? First, they swear, which is a very American trait. Most of theworld doesn't swear as much as Americans. They are “sick of” things. I statistically group
together pairs of words that show up together such as “sick of”, “my head”. And “tired of”,
“why do I”, “leave me”, these multiple word expressions give more clarity. In English, “sick”
means that you're not feeling well, but “sick of” means that you're tired of it, you don't like it.
So, more neurotic people are, the more they're swearing. But even they're alone and bored, theyrarely say the word of depression or depressed as it's not very socially acceptable to say I'm
feeling depressed. But if they do say it, they're more likely to be neurotic.
The opposite of being neurotic is being well adjusted (Figure 4). What do well adjusted
Americans talk about? As we know, somethings that Americans will say over and over are very
religious. You can see “the lord”, “god is good”, or “in Christ”, “the church”, “praise”, whichmeans in general, religious Americans are more well-adjusted, more agreeable. It's not so muchtrue in Europe, which is much less religious or China, or probably Japan, a few parts of theworld.
More NeuroticLess Neurotic

Figure 3

Figure 4

And what you can see are, “soccer”, “baseball”, “lakers”, words of sports. So, well-adjusted
Americans also talk a lot about sports, not just sports and “the gym”, where we know thatexercising is good for you. Exercising is almost as good for treating depression as taking
antidepressants. People who are watching sports are less neurotic. I have a hypothesis for thereason that is watching sports is often something you do with other people, a way to connect,

Page 4 of 22

and something you talk about to make connections. And we still need to do more studies tounderstand it, but it seems at least plausible is an interesting hypothesis. That involvement insports is a way of getting out of the house, meeting other people connecting for both introvertsand extroverts.
We can correlate how well our language predicts five factors of personality: openness,
conscientiousness, extraversion, agreeableness, neuroticism, comparing with how well a friend
will predict. If you have a friend on Facebook fill out the survey question for you, how good a
job can your friend does in estimating your personality from a questionnaire? And the answer
is the Facebook language computer model is just as good as your friend in terms of estimating
your personality. It’s not very accurate, but neither is your friend, still to tell something about
personality.
What does this say about well-being? We have a website called Authentic Happiness where
we've had about a million people come. On the website, they can take questionnaires, and they
can share Twitter if they wish. With their permission, we can correlate the language of their
tweets with what they said when they filled out their questionnaires (Figure 5). In America,
positivity is highly correlated with excitement. But we can see that now this doesn't have to be
the case. They can be happy, but calm and peaceful. Globally, positive relationships are one of
the two things that matter for well-being. What do people who score high on the surveys withpositive relationships? We can see “love”, “birthday”, “my”, “you”, “friends”, “family”, “baby”,“daddy”, which relate to relationships in America. They can be different across countries, but
you can see similarities. The other big component of well-being is meaning or accomplishment.
In America, but not in much of the world, lots of meaning comes from believing in “god”, andJesus, and having “blessings” and “faith”. And accomplishment comes from being “proud”,“winning”, doing a “great” job, and your “daughter” and “baby”, the amazing people you're
with. What do we see? We see positive emotion of love and being happy or positive
relationships, and meaning through religion in America, and accomplishment by being proudof both themselves and family, which are all showing up in language. So far, we've taken
surveys correlated with language, but it means now we can also look at your language in yourtext messages, in your tweets, and say something about your well-being.
Predicting Wellbeing: PERMA

Figure 5 (Word Well-Being Project: wwbp.org)

Page 5 of 22

We can also look at the other side (Figure 6), people who score low, the negative emotion of“sick”, “hate”, “tired of”, even swear words. Before these surveys, I thought that the word“people” would be a positive word or relationship word. But that's not the way it's used on
Twitter. Positive words of people are “friend”, “family”, “daughter”, “wife”. But negatively,
we can see “people are annoying”, “people are stupid”. “People” in English is a distancing word,
not a close word. People is under people, which means this is negative. Another example is theword, “I”. For example, if you're talking negatively, you might say “I am tired”, “I don't see
meaning in life”, or “I don't like people”. But positively, there should be much more ‘we,’ muchless ‘I’. That's one of the universal examples. Focusing on yourself is mostly not a good thing
to do for well-being, which is a measure of lack of well-being.
Predicting Wellbeing: Lack of PERMA

Figure 6 (Word Well-Being Project: wwbp.org)
Therapists often suggest that if you're depressed, go call up a friend to do something, and think
about someone besides yourself. We can see this in the language of lack of well-being. Here isanother example from the Authentic Happiness website. We have people take questionnaires tomeasure their character strengths, such as teamwork, leadership, kindness, and zest (Figure 7).
Predicting Wellbeing via Character Strengths

Figure 7 (Word Well-Being Project: wwbp.org)
Page 6 of 22

People who score high for teamwork, talk about the “future”, “office”, religion, and networking.
People who score high on leadership talk about managing things or “challenging”, doing“presentation”, who are “looking forward” and tend to travel more. Contrast that with different
character strengths, people who are high on kindness talk about their best friends, and they're
sorry to hear the bad things that have happened to someone else. They send hugs to people,
bring in cupcakes. They are people who are high on zest, thinking about wonderful in themorning and countdown until things are happening, and looking at beginning. We can see thesedifferent kinds of character strings in language, which is a way to measure.
The second part I want to talk about will be depression. I want to try and look at measuring
depression or well-being with the language people use. With the same way on Authentic
Happiness, we ask people permission to collect their Facebook posts, have them take a standard
depression measure with the questions that are asked by doctors or a psychologist to find thecorrelation (Figure 8). We've used a method called Latent Dirichlet Allocation, a topic method,that clusters words tend to go together automatically.
Depression

Figure 8
We see words indicating sadness, such as “crying”, “tears”, “broken heart”. We see words ofanger. We know depressed people tend to be angrier, more hostilities. And we can see the wordof “people” again, which is negative on Facebook and Twitter. We see words of loneliness,
which go a lot with depressed people tending to be lonely. We see the words of somatic
complaints, “sick”, “headache”, “body hurting”, all of which are known symptoms. These varyacross cultures. I've looked mostly at the US and China. In America, depression shows up lotswith depressed mood and loneliness where people tend to have more psychological aspects ofdepression. In China, people go to their doctor saying “my head hurts”, or “I'm sick”, “I'mhaving a headache”, “I'm feeling pain”, where they have more somatic body symptoms ofdepression. We can also ask as a third culture I haven't studied, for example, in Japan, which isdifferent from young people versus old people. And I see in India, this is changing. Young
Indians look more like Americans while old Indians look more like Chinese people. I worried
that if we go cross-culturally, and we develop all our skills in America, we forget about things
that are true for Americans but truer for Chinese and maybe other cultures. As for medical
references, we can see people talking about “hospital” and “pain”.
Let's try and understand more about what we can get in detail with different facets, pieces ofdepression (Figure 9). I'm going to show correlations of language with two sets of questions.
One set of questions about low mood, “I'm often in the dumps”, “I often have frequent moodswings”, or the opposite of “I seldom feel blue”. These are low mood measures. Then I will

Page 7 of 22

show measures of low self-worth, “I do not feel comfortable with myself”, “I am not pleased
with myself”, and “I dislike myself”. What we'll see in this language is that people don't say“I'm depressed”, or “I disliked myself”, but they use language to make it understood.
Depression Facets

Figure 9
We can ask what language do people use when they're sad or having low mood? Interestingly,
we can see that the most predictive word is “alone” (Figure 10). People on Facebook rarely say,
“I'm depressed”, or “I'm suffering from depression”, which shows grey as it's rare. If they do
say I'm depressed, then they do have low mood. But what do they say more? It’s “I'm alone”.
And they swear a little bit more it seems. Thus, we see this loneness, the symptom of depression,reflecting a socially acceptable way to express bad mood. Again, I worry that it is culturally
very American. But in America, you're not supposed to say “I'm depressed”, but it's okay to say“I'm alone”. which is a symptom of potentially in depression, along with lots of other words.
We can also ask low self-worth (Figure 11). For example, if I think I'm not a good person, what
do I say? I don't say “I'm a bad person”, but ask the word “why”, or “I don't feel this”, “I don't
believe this”. You can also see some of the introvert words of “actually”, “probably”. If you
have low self-worth, you tend to qualify with saying “I'm probably doing well”, “I'm actually
doing well”. These are ways of not really being confident. If you had high self-worth, in
America people like to brag much more. Primary American way people like to say, “I'm a good
person”, “I did a great job at my work.” If they say, “I probably did a good job in America,” itmeans that they probably don't think themselves as good. The same techniques can be used in
any language in any country.
Mood

Low Self-worth

Figure 10

Figure 11

Page 8 of 22

Next, we will look at empathy that’s not been studied so much but interesting for well-being.
We're interested in this because we're trying to build chatbots that are empathetic and big feeling.
Can we measure empathy, in other words, caring for other people? Can we understand it better?
We gave thousands of people a questionnaire to measure empathy then took their language into
analysis (Figure 12). We can see that empathetic people use the words such as “love”, “somuch”, and “heart”. In the word cloud, there's “love”, “soul”, and amazing “friends” and“family”, which are unconditional. Although we could also wonder, there's also a bit of “I” and“me,” which seems suspicious as empathy is about caring for other people, which means it’sme feeling you and it should be about you instead of me.
Empathy

Figure 12
However, in the book of Against Empathy1 written by Paul Bloom, a well-known psychologist
at Yale, he argues that you can measure both good (Beneficial) and bad (Depleting) empathy.
And that feeling for other people could be good and bad. How could that be? People probably
think that empathy should be good, and you should care about other people. But we think
there're two kinds of empathy. There's good empathy of thinking about other people, caring
about other people while there's bad empathy of feeling someone else's feelings. For example,
if I try and understand what your problems are and try to solve them, I should not get too excited
and worried about your problems. I should try and help you solve your problems, do things
well, which is good and means helping other people is good. In the other hand, if I'm a doctor,
and have to give you the shot which will hurt so much. Then I feel your pain. We want to beable to help and care about other people, but not to necessarily get overwhelmed with the painof other people. We now have a set of questions to measure good empathy and bad empathy,
beneficial and depleting. We can tease apart the components in language. If I often have tender,
Bloom Paul. 2018. Against Empathy: The Case for Rational Compassion. New York: an imprint of HarperCollins
Publishers.

1

Page 9 of 22

concerned feelings about people less fortunate than me, it is good empathy. And if I try to
imagine how other people feel, it is also good empathy. But if I tend to make other people
suffering my own, it should be bad empathy. So simply speaking, we have good empathy ofcaring about people, and bad empathy of feeling other people's pain.
Here's the American sample of beneficial and depleting empathy (Figure 13&14). For
beneficial empathy, we can see the words such as “wonderful”, “family”, “children”, and“blessed”, which are all good things. What's bad empathy look like? First, it is about “I” and“me”. I'm supposed to be concerned with you, but I've caught your pain. And now instead ofworrying about you, I'm feeling about “me”. We can also see “people” and “pain”. They arethings that may look good, “my sister” or “my friend”, and “the tears in the laughing”, but they
are also mood swings and depressed mood.
Beneficial Empathy

Depleting Empathy

Figure 13

Figure 14

Some things are good but much more complicated this empathy. For example, if you are atherapist or a doctor, or any helping profession, and when you have too much feeling someone
else's pain, you burn out, get tired and you can't do it anymore. You can't go in and deal withpeople dying of cancer every day and feel their pain. If you feel the pain of every cancer patient,you end up feeling about “I” and “me”, which is not good. We can tease apart these pieces.
As showed in Figure 15, we can also ask these same people survey questions, people withcompassion, the good empathy, less time volunteering, more time giving money to charity, lessmissing work, less drinking on work nights, which mostly is not healthy and less stress. People
who feel other people's pain, more time volunteering to help. That's maybe good but doing it
less giving money, more missing work, more drinking at night, worse health and much morestressed. And if you're in a stressful job dealing with people's pain, if you're a psychotherapist,
if you're a doctor, you really want to be over on the compassion, caring, not on the feelingsomeone's pain.
Empathy as Revealed in Behavior(Controlling: age, gender, beneficial /depleting)Figure 15
Page 10 of 22

Last, I want to shift to talk about measuring whole regions as measuring individuals has beenmuch discussed. The US is broken up into about 3000 counties. About 2000 of them haveenough people to actually measure data. Many of them are almost empty farmland. America isvery big and very empty. So, it's about a sample size of 2000 counties. We collected Twitterfrom public pieces and mapped it to the county. Then we take data collected by the government,
the Centers for Disease Control, our main Medical Group, or the US Department of Commerce,
Department of Census, which does a survey every 10 years. We have government collected
data on health outcomes and well-being outcomes, life satisfaction, Gallup surveys, and otherones in the government. We can now take the language used in the county on Twitter andsatisfaction with life, or mortality from heart disease. We can see how counties differ in theirlanguage based on how happy they are, or how healthy they are.
I've shifted to the county level. These are the topics, clusters of words that most correlate withwell-being across the US counties (Figure 16). Two of them in red at the bottom are counties
with low well-being. These are counties that are bored and tired. What are counties talking
about with high well-being? “Personal training”, “meeting”, “conferences”, “suggestions”,“ideas”, “donating money”, counties talking about these are wealthier, but especially within thewealthy counties, the ones that talk more about “donating money”, “information”,“management”, “good jobs”, “building”, “compassion”, “caring about people”, “experiences”,
and especially like going to the “ocean”, going to the “mountains”, going “camping” on theweekend. These are communities of Americans that have more well-being, and they are doing
more outdoor things in nature. They are working with other people and donating more money.
Life Satisfaction across Counties

Figure 16
The other piece for the last topic is that well-being ties to health. The question is, can we actually
measure something about language and health outcomes? Again, we'll take the Twitter
language, then map it to a county, and correlate it with mortality from heart disease, dying fromarteriosclerotic heart disease, hardening of the arteries (Figure 17&18 2). We can see in counties
with more heart disease, there's more hate and interpersonal tension. And there's more swearing,anger, hostility, more boredom, and tiredness. So, if your neighbors are being negative andswearing, you're more likely to die of heart disease. Lower heart disease on the positive side,people are talking about “conferences”, “customer”, “management” (skilled occupations).

2

Eichstaedt JC, Schwartz HA, Kern ML et al (2015), Psychological Language on Twitter Predicts County-Level
Heart Disease Mortality, Psychological Science, Vol.26(2), p163.

Page 11 of 22

People are talking about “great food”, “friends”, and “fantastic weekend” (positive experiences),and they are also talking about “opportunities”, “overcoming” problems (optimism).
Associated with Lower Heart Disease

Associated with Higher Heart Disease

Figure 17

Figure 18

How does the language do well to predict heart disease? We can look at heart disease rates asreported on death certificates, and we can compare that to our predictions (Figure 20 3). They
are similar.
Predicting Heart Disease

Figure 20
In Figure 21, we compare the predictions of AHD mortality from regression models withdifferent independent variables. Predictive performance was slightly but significantly better fora model combining Twitter and the 10 traditional demographic, SES, and health predictors thanfor a model that included only the 10 traditional predictors 4. What we can see is that males havemore heart disease that predicts a bit. Smoking, diabetes, hypertension, obesity, all of thempredicted dying of heart disease, which is not surprising. Twitter captures so much about thecommunity that it knows not just education and income but lots of behaviors of the culture of
the community. We get strong signal here. We use similar sorts of measures to look at things.
For example, voting, where we find that, who is it that switches, which counties switched tovote for Trump, ones that were pessimistic about the future. In areas where people think thatEichstaedt JC, Schwartz HA, Kern ML et al (2015), Psychological Language on Twitter Predicts County-Level
Heart Disease Mortality, Psychological Science, Vol.26(2), p165.
4
Eichstaedt JC, Schwartz HA, Kern ML et al (2015), Psychological Language on Twitter Predicts County-Level
Heart Disease Mortality, Psychological Science, Vol.26(2), p162-164.
3

Page 12 of 22

the future looks bad, they vote to change the government, which is not surprising. Also, if
they're poorer, but that's less predictive than pessimism about the future.
Predicting Heart Disease

Figure 21
Also, I would like to talk about what else I'm working on for possible joint work in the future.
The biggest thing I'm focusing on is to measure cultural variations across different cultures, andto try and understand how happiness is or isn't the same cross cultures. For example, Japanesepeople are on average less happy than Americans given that Japan is a wealthy and well-run
stable government. You expect unhappiness in Africa or someplace with terrible problems, andJapan is a very successful country, a wealthy country, where people are less happy. Why isthat? I don't know. Maybe because people put less importance on being happy and more
importance on helping their family and helping their neighbors.
I was talking right before we started that Americans are twice as lonely as Japanese. America
has a much more individualistic culture than Japan. We tend to do our own things and try to be
happy, which is mostly not good. You should try to have good relationships and have meaningin life. Trying to be happy is the wrong way. But it's an interesting question. What do Japanesepeople value? Is depression the same? Do symptoms show up the same?
In America, people like to say, “I'm so good” or “I'm proud”. And Americans hate saying “I'mashamed of doing this”. In Japan, I'm told by my collaborators, people don't like saying, “I'mso good.” or “I'm proud of how smart I am”. They do like saying, “I'm so ashamed” or “I didn't
do a better job”. It is very different of what's acceptable in the culture. In America, higher
arousal and more excitement is good. We want to be positive, and we want to be excited andpositive, but maybe calm is good. Being satisfied and content is good and being excited is lessimportant than America. But I think to measure these things, which are minor technical details.
Most people in Japan speak Japanese. Most people in China speak Mandarin. Most people in
America speak English, although we speak lots of languages. That's uninteresting. Moreinteresting is what do people care about. Are they caring about taking care of their parents, theirchildren, or their co-workers? What is their value? And what is it they want? What does wellbeing mean to them? And then can we measure it in ways like using language? That's mybiggest piece.
The other piece is ChatGPT. In America, everybody just talks about ChatGPT, or these models.
ChatGPT is a new product, which has in the last month a million new users. You can type any
question. For example, I typed right before this meeting, “please translate the following into

Page 13 of 22

Japanese”. And you can see the translation. The question is, how might I use ChatGPT toincrease well-being?
Comment and Q&A
Prof. Masaya Kobayashi
My present question is two points. You very clearly explained your main works between
positive psychology and social media inquiry or text analysis. The point is that, obviously,
positive psychology or such kind of psychological investigation have clarified various points
to the world, but I wonder what is the new point which Twitter analysis or social media analysis,
takes analysis clarified the point, which was not found by questionnaires? Some of your present
talk like empathy is very fresh to me. I suppose such kind of arguments or new findings arebeing already done by your team and such kind of investigation. That's the first question.
Second, I am very interested in your research on county level or community level because
county/community is quite related to social or political dimension. Also, I have known that yourteam has already done great work on the election through social media language and inquiry.
That's one of my priorities of our research team because I have already made some progress inmaking research on the relation between Japanese election and positive psychology. Could you
somehow briefly introduce your findings to Japanese audience because it will facilitate our
project team. I hope that we can collaborate with you on this point.
Also, I would like to hear from your frontiers research on cultural variation. I am very happy if
you could explain what the present stage of international collaboration and cultural variation is.
Prof. Lyle Ungar
The first question, what do we get from social media that we don't get from questionnaires? I
think there are two answers. One is for a questionnaire. You need to know what to ask. If you
know what to ask, you get a good answer. If you don't, then you don't. For example, when I
looked at well-adjusted Americans, I was surprised to see how much people talked about sports.
Now, I know that sports is very important in many cultures. I know because I see that Japanshows up in the sports news here, too. But I hadn't thought of, and people had not thought ofsports as being part of well-being. People think about exercise as being part of well-being, butthey don't think much about sports. That for me was an interesting hypothesis generation. And
it's common in the marketing world to have focus groups. You pulled together 20 people, you
paid them to ask questions, for example, how do you feel about something? A car, a product, a
politician. In some sense, what I get on Twitter or Facebook, a very large focus group. It's nota random sample of people. But it's a bunch of people. What do they say? What do they say
about this? What are the comments about this? If you want to know what people think about
GE or Biden, then Twitter is not a representative sample. But it's very cheap, very fast changing.
What are the reactions to this? For example, we're seeing a lot of press now in France because
they're trying to raise the retirement age. Lots of protests there. Macron is worried because
there's a lot of contests there. He thinks he has the right solution that people are getting older,
they should work longer. Other people are burning stuff in the streets. Can we do a very quick
understanding of what people are arguing about in France by monitoring what they're saying?
Does that make sense? It's so much more expensive to send pollsters out and ask questions.
And it's not even clear what the questions are that you should ask. Why are these people? From
my perspective, I didn't understand it until I talked more to French people. Americans don't feelso negative about working longer. We'd all like to retire early, not me. I like working. But
there's something different about friends. There's a belief in welfare state. I think one thing is
to get answers quickly to questions you didn't think to answer.

Page 14 of 22

The other piece is that polling is very expensive. And in the US, you can get detailed polls at
the state level. But the counties are expensive. It's hard to have details. Not surprisingly, we
often don't know the outcome of an election until after they count the votes. I always think thisis crazy. We're spending tens of millions of dollars on polls. And we don't even know what
President the county will vote for, let alone more detailed questions are what people care about,what they're happy or not about. It's much cheaper to measure Twitter. We can't get tiny
counties, but for 2000 counties, we can get enough tweets to at least see if people are talking
more about this or less about this. What you get is an ability that's much finer grained. It's not
the same quality as a Gallup poll. I love the Gallup polls. They're very good. I love the US
Census. They're very good. We have a very good Census Department. But it's too expensive to
ask everybody every question. And even things that I think are incredibly important. How
lonely are American? No one has done a good poll across America to measure loneliness.
People, insurance companies, health companies have asked small sample, 100,000 people, but
not enough to have a representative sample, and not to know how much for sure as it's varying.
Not to know the age effects, lonely people. The loneliest people are mostly people 18 to 25,
which is somewhat surprising, or people over 80 who can't leave their house. Can we get thedetails? How is that changing? So, note that there are lots of things you wish you could do apoll on, but it's expensive. And if you want resolution, how is it different in the cities versus thecountryside? Many countries, I don't know about Japan, but I know about China and the US,
the cities and the countryside are different worlds. There are different incomes, different
concerns, and different social relations.
We need to have measurements that kick these things into account to just group a whole
prefecture together with the city and the surrounding countryside. Again, I wish I could do polls,
but they're too expensive.
Second question about county level politics. I think the question we've asked mostly is, whatcauses people to try and vote out the incumbent? In general, if you're in power, you stay inpower unless somebody is unhappy about you. And now the question is more precisely, whatforms of unhappiness do people have? What do people care about? The classic economic
argument is people care about how much money they have. And if the job market is bad, andthey can't find a job, then they vote for change. And if the job market is good, and they have ajob, and they're making money, they stay. This is not a bad approximation. There's been betterstudies in Europe than America, because Europe is a whole bunch of countries, you get moredata. You can look across countries, lots of elections. This is true. The economists are right.
But it's a little more subtle. And what we see in America, and I think we will see also in Europe,is what really matters is how optimistic people are about the future. In America now, things arevery strange. People are mostly thinking they are doing okay, job is fine, but the country isgoing the wrong direction. We're going to fall apart. There's too much descent. For example,
the economy's unstable, and AI, or China's going steal our jobs. Mostly people worry about
China. They should be worrying about AI, but they're still insecure. And note that even though
they're currently having a good job, unemployment is very low right now in America. But
people are worried. And when they're worried about the future, they vote for change. So, notethat this is now a psychological change that says that people in economics did the first pass,which was measured dollars, or Yen, or RMB. But that's not the bottom line that makes themetaphor. What people really care about is not just the money, but how well things will bedoing. And now we have a more psychological basis, which ties to money about the future.
And that I think is interesting. And there're lots of room across different countries to measure,what are the real psychological factors that drive elections. And the way that we looked atmostly is, who votes for incumbents, who changes their vote, who used to vote for one party

Page 15 of 22

but now votes for the other one. America had a very big shift when Trump was elected. Many
people were unhappy about the way America was going. And they voted for radical crazychange. Some of them wanted to vote for Bernie Sanders, a socialist with radical change. But
the majority went to Donald Trump, which was also a radical change. Maybe too radical. Manyof us didn't like it. But I have a more optimistic feeling about America. And many people don't
love Biden, but if you think, “Oh, things are going okay, keep the guy. He's okay.” That's a
psychological story and one that I think needs lots of work over the coming years to say, what
are these big psychological components that vote and drive voting, and how can we measure
them?
Finally, about cultural variation. For the Japan side, Jeanne Tsai at Stanford, a cultural
psychologist, who has collaborated as she started in China, but now also in Japan, has done anumber of studies on “ideal affect”. She's a close collaborator and has collected lots of tweets.
And her theory is that in America, people want positive, high energy, excitement, high valence,high arousal. If you violate that by being negative, you get more press. And in Japan, people
want positive, lower arousal, you want to be calm and positive. If you violate that by being
more excited, you get more retweets. She calls that “ideal affect”. These are how the culture
wants to be perceived. She's now collecting not just general tweets in Japan but also commentson news articles. What comments on news articles in Japan get more attention? Her hypothesis,
which we haven't tested is, if it's more excited and positive, it'll get more attention, and if it's
called, whereas, in America, being negative, will get more attention. It’s a question of whatdraws attention. And in some sense, if you violate the cultural norms, do something different,
then that draws attention.
And I am trying to look at questions like less political shame. In America, talking about shame
violates the norms. In Japan, you may feel bad being ashamed, but you're much happier to talkabout it. And this to me is interesting in terms of what one wants to be and how you talk. If you
do surveys, you can see Japanese say they are less conscientious than Americans say we are.
This is ridiculous. You started the meeting exactly at the top of the hour. If this were America,
would have been 3 to 5 minutes late. If you look at behavioral measures, Japanese are very
conscientious because people show up on time and they do what they're saying to do. America
is rather more relaxed. Brazil, Mexico is way more relaxed. But the surveys don't confirm that.
What's happening? I think when you ask a Japanese person, “are you conscientious?” He looks
around friends and coworkers and say, “well, not as much as he is, he's much better. Plus, I
should be modest. It's not good to say how conscientious I am”. If you look at Americans, welook at other Americans and say, “Ah, yeah, compared to him, I'm only 5 minutes late. I only
missed one meeting last week. I'm very conscientious. Plus, I'm in America, and it's good tosee how good I am.” So, note that having behavioral measures often gives you something thatyou don't get from surveys. And my final point, language is more of a behavioral measure thana survey. My hope is that you may actually see some language what people are doing, not just
how they think they should answer the questionnaire.
Prof. Masaya Kobayashi
From my point of view, your research on the macro level is very beneficial for Japanese, notonly researchers, but policymakers and governments for considering what to do in terms ofwell-being. Is there any example those public institutions have already used your kind ofanalysis? Also, I have already read your articles on the Trump election. I wonder if there is any
development after election.
Prof. Lyle Ungar

Page 16 of 22

Let me answer them in reverse order. No on the Trump, but there have been nice studies in
Europe, by my collaborators. In terms of government agencies using it, in the US, only the stateof Washington was very interested during COVID and trying to monitor the well-being andhow it changed. COVID had a huge negative effect on the population. The question is, can wemeasure the psychological well-being of the people in the state? And what are their concernswith respect to COVID? What are they worrying about? We've also been talking to the Census
Department of Mexico. They're not using it yet, but they're very interested. Mexico has way
less money for doing census measurements than America does. It’s not as wealthier country as
US or Japan, obviously. And their question is if they can do some of the measurements thatthey wished they could afford to do. If they were America or Japan, they would do more surveys,
but they're not as wealthy. They're trying to use Twitter. They've used it for some things to track
tourism flow during COVID. As you can see, by loading where the English language tweets
are travelling, where tourists are, so it’s way of monitoring tourism, which as you would haveguessed, has had a big hit during COVID. And it's a much bigger part of the economy there.
But that's the sort of piece where they're moving toward is if they can find big signals, forexample, tourist location, or big shifts in well-being due to exogenous shocks like COVID.
Prof. Hikari Ishido
I was thinking what the cultural difference between US and Japan in terms of could be
responding to this question of whether you are happy or you think yourself as a conscientious
person. My way of thinking is that US society is kind of like an open loop thinking society.
You can just speak out what you think directly, because when you have a negative repercussion,
you always have this new frontier, new cities of escape and everything. But here in Japan as anisland and high-density population country, we really don't have escape cities to go to. We haveto anticipate this envious repercussion, negative kind of repercussion. So, in anticipation ofthese negative feedbacks or repercussions, our initial response is already cautious or understated.
Technically speaking, are there any sort of statistical information technology technique
dissecting between initial, I mean, direct speaking up component and preventive, or cautious
repercussion, or anticipating type of understatement? Basically, causality can be unidirectional.
But in the case of this closed loop, “what you're thinking”, the causality could be bi-directional,“what I'm talking”. So, when I say, “Okay, I'm a conscientious person”, then I would alreadyanticipate a negative or envious repercussion from this high population density community. I
will say, “No, I'm not conscientious as my colleague”. It’s understatement. Although I mightsay, “Okay, I started this session on time”, and I woke up. Any statistical or IT technology
which can dissect these two impacts?
Prof. Lyle Ungar
Let me start with a little background, which we're looking at. The cultural psychologists
distinguished individualistic societies and collectivist. The name that they'd apply to exactly
your story is that Japan is much more collectivist. And collectivist society on the one hand, youtake care of your family and your parents and your neighbors, and the other piece, you're also
stuck with the same people forever. This is your small island. In America, if you don't likesomeone, just go away. We don't believe in having enemies like some countries do. Either
you're my friend, or I don't talk to you. We have this sort of non-attachment for better or forworse. In a collectivist society, it's your mother, what do you mean, you're not going to talk toher? It's your responsibility as a son to take care of her. This connection that is always there,
and because of this inescapable connection, relationship management is central, more important
than anything else. That's the first piece. Now the question of, is there IT ways to do this? I

Page 17 of 22

don't know. Let me shift now to China, which I've looked at more, which is also a collectivist
society compared to the US. What Chinese people say is that understanding what said in any
given statement in Chinese is much harder than English. In the individualistic English language,
each sentence is self-contained. It's a low context language and a low context culture. Whenyou say something in America which is a country of immigrants, you need to be able to talk tosomeone. If you look at my research team, there are German, Chinese, and Surinamese. They
all speak different languages. That's a very international group. It's very important in that culturethat American culture to be very clear and direct. A collectivist culture life is high context.
When you talk to someone, you know something or if not, you want to have dinner, you wantto meet their parents, you want to meet their. It's knowing the background and so when you say
a given statement in a high context culture, the information is not in the statement. You need toknow who you're talking to, you need to know the background. This is the psychology story.
Now the question, which I have not yet done is, we should be able to actually show this withour computer models. I haven't done this. It should be case that many Americans statements,
computer reads the statement, and the meaning is clear.
In a Chinese statement, and I think Japan will be very similar, Japan as a collectivist, highcontext culture, to understand the statement, will require knowing more context. That meansthat in some sense that things like ChatGPT, which is a low context American product. It doesn't
know who you are. t doesn't know what your relationship. It doesn't know what you care about.
It doesn't know who’re your parents. All it knows is the statement you put in. It's a very low
context. There should be computable measures of how well I can estimate anything about
someone but their intent, concern, feeling, and beliefs, should be harder in Japanese than it isin English, if you believe that theory. So, that's my hypothesis, but I have not tested it yet, which
we should.
Mr. Alfonso Torrero
I was thinking about some practical applications of your research for having a fairer society. I
thought that identifying words and sentences related to personality traits such as neuroticism,
depression, could be useful to show specific health apps in people's social media. Instead ofthat we are used to get ads from products and services that we search for, but maybe related tothis group, it could be very useful to relate the words the sentences that you have found to show
people some psychiatric help, or maybe specific social groups. That's what I thought.
Prof. Lyle Ungar
I would love to have the ability to both provide people, not psychiatric care, which is toocomplicated to do by chatbot but at least help and advice to make this available. Most people
have cell phones now. We can give them access to lots of information in a way that's more chatbased. The other piece I would love is, as you say, not to sell people products, which theymostly don't need, but to have something that's much more of a targeted. For example,“Wouldn't you like to go for a walk now?” I see based on your schedule, that you have a halfhour until your next meeting. Or last time that you had lunch, you enjoyed it with this person.
Why do you haven't put lunch on your calendar for a while. Why don't you call this person up?
I'd love to see ads, or chats, or cheap communications that encourage healthy behaviors. But
for me, I'm much less interested in telling people not to smoke. They all know smoking is badfor you. But to help them to remind themselves of, here's how to take the time to go for a walk,there is the time to talk to your friend. Here's compliment someone. When you're in the shop,
say something nice to the shopkeeper. There are million things you could sell people that are
free behaviors that do not cost money but make people more satisfied and happier with theirPage 18 of 22

lives. And we don't have a good infrastructure. I think the fact that everyone pretty much has
cell phones now means that if we can figure out how to target these in a way that's useful, rather
than annoying, where people opt in and say, “Yes, this is what I want to do. This is my goal.”
They say I wanted.
I'm hoping we can help build chatbots that target them at the time they want, in the language
they want, with the frequency they want, gives them helpful reminders that they want. I think
the cost of these things coming down is great. And the benefits should not go only to Google,
Amazon, Facebook, or TikTok, which mostly does do not make people happier. It should go toactually increasing well-being. And if the cost comes low enough, it means that you could
almost have a personal trainer, which is a wealthy person luxury, available for free on yourphone, which should be something affordable to everybody to help them be motivated toexercise. My fantasy is that we really have very cheap interventions that people can say, “Thisis my goal. I want to walk twice a week half hour”, and the app helps them do it. That's ournext big mission is to start building these apps and testing them.
Dr. Xiaofang Zhang
I am Chinese, but now I live and work in Japan. You said that your team also analyzed Chinese
well-being. About Chinese well-being, I have two questions for you. First, as a Chinese, I'mvery interested in which social media your team used to analyze Chinese well-being, because
as you know, Chinese does not use Facebook, Twitter, or Instagram, that kind of theinternational social media much. So, which social media does your team focus on in China?
This is my first question. My second question is, in the well-being analysis about Chinese
people, which result impress you?
Prof. Lyle Ungar
I would like to use WeChat. But the security is much too tight. Facebook is very easy to get,and everybody can share things. WeChat has very good security. Even if people want to share
their WeChat with me, it's not easy for them to do that. But Weibo is easy for people to do. And
we use the same sort of techniques in China we do in America. That is there lots of market
research firms. And what market research firms do is they pay people to take questionnaires
and to share their posts. The same thing that people use who are trying to understand products,
the Chinese markets use, we use to understand to try and measure well-being. That part isrelatively easy, and it's controlled. It's tricky that you can't ask questions about political
government which I'm not interested in. And there are some tricks that make it easier because
China is very happy with Hong Kong being ambiguous. Hong Kong is part of China which also
tends to support many of the American companies like Amazon Web Services. And so, we canrun servers in Hong Kong that are both in China. And so, we're allowed to check Weibo andare accessible to the American infrastructure. You don't have to have as many Chinese accounts.
That was relatively easy when it’s always problems with data sharing.
What have I found? Not much yet. We're just starting. We started looking at not well- being
but individualism versus collectivism. And everybody talks about America versus China.
America is individualistic and China is collectivist. China's a very big country. And there aresubstantial differences between the regions that were historically wheat growing Northern and
the Southern that are historically rice growing. This is also true within Japan. We replicatedthese findings in Japan.
You can look at different regions where 50 or 100 years were mostly wheat and mostly rice,
both in China and in Japan, then collect Weibo in China, Twitter in Japan, and we now look at
the language, look at how people are talking. And there are substantial differences. And you

Page 19 of 22

get in the collectivist region's much more small community. In the individualist, both things I
would have thought more consumer goods buying, and we're talking about Beijing, and things
that are international. My first thought was, why is that? But collectivist is your local
community. In China, that's not a collectivist view. Collectivist, it is your community, whereas
the more individualistic parts of China are more engaged with the nation.
The other piece we're starting to look at is urban rural. There's a huge difference and starting to
see what some of the concerns of people are in urban versus rural regions. And most of them, I
think, are obvious to anybody in China. But I haven't realized as an American, how challenging
it is in the large city to find an apartment for your parents that you need to take care of them,and real estate is very expensive. And it's a trivial finding.
But I think in terms of these questions of where this is going, I think that if you think about
governments and people concerned about community, trying to see what people are worrying
about, this is something where one could do a more detailed analysis of how much is the
Shanghai versus Beijing versus the smaller cities. What are the different concerns? We havenot looked in detail there, but I think that's sort of the direction is to try and understand more
about what do people in different groups. We're also trying to understand stress. What arepeople stressed about? Emotionally, it's the obvious. China's a very exam-oriented country, notto say that Japan and the US are not also exam-oriented. But we see a lot more concern about
exams and a lot more concern about rapidly changing, rapidly growing wealthy country. I missthe rise there. The answer is now I don't have anything good to report. But we're still in the firststages of setting up the infrastructure to do some surveys to quality control it because lots ofpeople just do the surveys for money and don't really answer honestly. And to then collectWeibo and do some more analysis to very early results.

Assoc. Prof. Xiang Li

I did some text mining before. I really feel your research is very useful and has very bright
future application. I'm just wondering, the knowledge you mentioned can be applied to thehuman behavior prediction. For example, you talk about the emotions, and five big emotions
characteristics. Maybe that kind of knowledge can be used or applied in human decision andprediction. For example, how people decide to shop, how people decide to make certain
behavior. I'm just wondering if that is also possible to do that.

Prof. Lyle Ungar
People have done some targeted advertising. If you're an extrovert, show an ad with you, and10 people playing together, show it as something with a big crowd. If you're an introvert, sellsomeone a game that you play by yourself, something you can do by yourself. There is somework there. Mostly, personality is not so useful for predicting marketing as what you havebought in the past. The big companies mostly use what you clicked on, what you have boughtin the past, better than most of the language. People have tried in Britain. They tried selling car
insurance. If you have a conscientious personality from your Facebook, we'll give you a
discount, because we think you drive better. But that only lasted a few days because people
thought that was very intrusive and violating privacy, even though people volunteered to share
their Facebook. I think one thing to realize is there's big cultural differences in terms of how
comfortable people are in sharing.
I think in America, people are much more willing to share their social media. In Britain andGermany, people are more private. They think that it's not right for a company to access theirsocial media. I think there's a big question here of consent. I think there are big differences
across cultures, in terms of how happy people are to have their information being used. I also

Page 20 of 22

think that if people know their information is being used to help them, they might be morepositive. But it's still unclear to me when these things are used, meant where they're helpful.
I've also thought less about selling things. I'm a positive psychologist. I want people to havemeaning in their life and good relationships. I'm less interested in selling them stuff. But I think
many of the same methods are used. I think the open question is, for predicting behavior often
measuring behavior is the best prediction? What are people buying? Where are they going?
Who are they talking to? I think one wants to have not just language but actual behavioral
measures. And what I see companies doing more and more is using actual behaviors, what theyknow from your phone, what they know from your shopping, and less of the language. But it's
had some signal.
Prof. Masaya Kobayashi
I wonder whether there is any influence of recent change of management on social media suchas Twitter or Facebook. Do you think there is any impact or influence on your research?
Prof. Lyle Ungar
Facebook, no and Twitter, yes. It was free to get 1% of all the tweets in the world. For decade,we collected 1% of all the Twitter, and sometimes more, which is very cheap. Now, it's become
tremendously expensive. Twitter is now charging $10,000 per month. Currently, there is no
academic discount. That's now $100,000 US a year, which is a lot of money to spend to collectdata. It's really going to make a huge difference. I think if this happens, it's going to be very
hard to do well-being research or political science research because the cost is just prohibitive.
If you're a company like Microsoft, 100,000 a year is nothing. But for academic, this is a large
budget. I think it's unclear how that's going to go. I think we're thinking about changing to havemore people be willing to let us to text messages and communicate by text message. I think it'san open question.
We look at Reddit, where people post things. Twitter is really global and everywhere in theworld uses Twitter. I think there's a big open question now of how we can cost effectively
collect data. And maybe, it's going to shift away from monitoring Twitter, toward measuringcomments and news sites, which we can grab in the US and Japan, where people have comment
pages where they do things, but that's a very odd set of people. Most people in the world don't
actually bother to read news articles and comment on them. Maybe, it's something that involves
much more open-ended polling. One thing we've looked at is asking people just questions. For
example, “How do you feel about the economy?” And just take the text, and we can process
that, which is alternative from asking a fixed polling question because I want open-ended
questions.
It’s a period of uncertainty, but I think that Twitter may change their mind and decide that theycare about supporting well-being. Or they may decide that they just want to make money, andthey don't care about helping. It doesn't cost them a lot to give us the tweets. And in some sense,
it benefits them because we then publish methodology companies can use. But clearly ElonMusk's current concern is not sponsoring academics, but how to make money tomorrow. And
it's going to be a big problem for that measurement. I think we are looking for other ways tocollect data.
Prof. Masaya Kobayashi
I wonder whether there are any differences of analytical predictivity between using Twitter orusing Facebook. Is there any difference in the field?

Page 21 of 22

Prof. Lyle Ungar
The methodology is very much the same. The topics covered are different. We've foundsurprisingly that we do roughly as well as estimating depression from people's Twitter or fromtheir Facebook. We have people where we've collected both from them and then measures,closer than I would have thought. Surprisingly, there's a fair amount of political discussion on
both Twitter and Facebook. Amazing number of Americans get their news on Facebook. We
used to measure influence by looking at what newspapers were read in a given community.
Americans don't read newspapers anymore mostly. Everything is online. It's Facebook,YouTube, or Tik Tok. And so, it's harder to measure it. But all these media have lots of pieces.
One thing we're thinking about TikTok, at least in America is enormous. And TikTok isproviding APIs to download TikTok. It's much more expensive to process because this is audio
visual rather than text. Twitter is cheap. I can afford to process them on a university computerthat I own, or university owns. For TikTok, I'm going to need more compute power. But I thinkthat's the way things are going is more of this multimedia piece there. And people arecommenting and watching TikTok. It’s measuring something. I think that's the next question is,
how we do process in these other pieces where it's much less obvious.

Page 22 of 22