Package: edu-convokit
Edu-ConvoKit: An Open-Source Framework for Education Conversation DataTheEdu-ConvoKitis an open-source framework designed to facilitate the study of conversation language data in educational settings.
It provides a practical and efficient pipeline for essential tasks such as text pre-processing, annotation, and analysis, tailored to meet the needs of researchers and developers.
This toolkit aims to enhance the accessibility and reproducibility of educational language data analysis, as well as advance both natural language processing (NLP) and education research.
By simplifying these key operations, the Edu-ConvoKit supports the efficient exploration and interpretation of text data in education.📖 Table of ContentsInstallation|Tutorials|Example Usage|Documentation|Papers with Edu-ConvoKit|Citation|ContactInstallationYou can installedu-convokitwith pip:pipinstalledu-convokitOverview of theedu-convokitPipelineTheEdu-ConvoKitpipeline consists of three key modules:preprocess,annotate, andanalyze.
The pipeline is designed to be modular, so you can use any combination of these modules to suit your needs.
This pipeline has also been used by prior work in the field of education, so you can easily reproduce their results.TutorialsWe have provided a series of tutorials to help you get started with theedu-convokit.Basics ofedu-convokitThere are three key modules of theedu-convokitpipeline:preprocess,annotate, andanalyze.Tutorial: Text Pre-processingTutorial: AnnotationTutorial: Analysis...Datasets withedu-convokitWe've applied theedu-convokitto a variety of datasets. Here are some examples:Tutorial: NCTE DatasetTutorial: Amber DatasetTutorial: Talk Moves DatasetExample UsageContents:Pre-ProcessingAnnotationAnalysisQualitative AnalysisQuantitative AnalysisLexical AnalysisTemporal AnalysisGPT AnalysisPre-ProcessingThepreprocessmodule provides a set of tools for cleaning and formatting raw text data. Text pre-processing is a critical step in handling education language data.It ensures the data is clean (education data is notoriously messy).It ensures the data is standardized, ready for annotation and analysis.It ensures that the students and educators are anonymized; this is important to protect the privacy of individuals involved and allow for safe secondary data analysis.Here's an example of usingpreprocessto anonymize the dataset with known names:>>fromedu_toolkit.preprocessorsimportTextPreprocessor# For helping us flexibly load data>>fromedu_toolkitimportutils# First get the data>>!wget"https://raw.githubusercontent.com/rosewang2008/edu-toolkit/master/data/talkmoves/Boats and Fish 2_Grade 4.xlsx">>data_fname="Boats and Fish 2_Grade 4.xlsx">>df=utils.load_data(data_fname)# Handles loading data from different file types including: .csv, .xlsx, .json# Show some lines that contain names in the speaker and text columns.>>df.iloc[25:35]💡 Note: We see that the names occur in the speaker and text column.e.g., names like David and Meredith appear in the speaker and text column.The teacher is always shortened to “T” in the speaker column.We can use theTextPreprocessorto anonymize the data in both columns.# Creating variables for the columns we want to use>>TEXT_COLUMN="Sentence">>SPEAKER_COLUMN="Speaker"# Show the names of the speakers. In your use case, you might load this from a file or database.>>print(df[SPEAKER_COLUMN].unique())['T''David''Meredith''Beth''Meredith and David''T 2']# Create list of names and replacement names. We will make the replacement names unique so that we can easily find them later.>>known_names=["David","Meredith","Beth"]>>known_replacement_names=[f"[STUDENT_{i}]"foriinrange(len(known_names))]>>print(known_replacement_names)['[STUDENT_0]','[STUDENT_1]','[STUDENT_2]']# Now let's anonymize the names in the text!>>processor=TextPreprocessor()>>df=processor.anonymize_known_names(df=df,text_column=TEXT_COLUMN,names=known_names,replacement_names=known_replacement_names,# We will directly replace the names in the text column.# If you want to keep the original text, you can set `target_text_column` to a new column name.target_text_column=TEXT_COLUMN)>>df.iloc[25:35]💡 Note: Nice, we can see that the text has been anonymized (e.g., line 31)! Now let's anonymize the names in the speaker column.df=processor.anonymize_known_names(df=df,text_column=SPEAKER_COLUMN,names=known_names,replacement_names=known_replacement_names,target_text_column=SPEAKER_COLUMN)df.iloc[25:35]🎉 Great, now we have anonymized the speaker names as well! Some other great things are that:We have a record of the original names and the anonymized names. So if we want to go back to the original names, we can do that.The anonymized names are consistent: So [STUDENT_0] in the SPEAKER_COLUMN will refer to the same [STUDENT_0] in the TEXT_COLUMN.We can also use theTextPreprocessorto group the utterances from the same speaker together.df=processor.merge_utterances_from_same_speaker(df=df,text_column=TEXT_COLUMN,speaker_column=SPEAKER_COLUMN,# We're going to directly replace the text in the text column.target_text_column=TEXT_COLUMN)df.iloc[25:35]AnnotationAnnotation is a critical step in understanding your data, and it is important to do it right and consistently across datasets.
Annotation is useful because:It creates descriptive statistics about your data, which can help you understand the data.It quantifies the language used by your students and educators, which can help you understand the language.It measures the interaction between the student and the educator, which can help you understand the interaction.edu-convokit is designed to support these purposes with theannotatormodule.
Here's an example of usingannotatorto annotate the dataset for talktime, student reasoning, and teacher conversational uptake.
We're going to use the same dataframe from the previous example; sodfwill be the dataframe with anonymized names and merged utterances.>>fromedu_convokit.annotationimportAnnotator>>annotator=Annotator()# The talktime values will be populated in this column>>TALK_TIME_COLUMN="talktime">>df=annotator.get_talktime(df=df,text_column=TEXT_COLUMN,analysis_unit="words",output_column=TALK_TIME_COLUMN)df.head()🎉 We can see with a single function call, we’ve added our first annotation – talktime – to our data!Let's do the same for student reasoning and teacher conversational uptake.# The reasoning annotations will be populated in this column>>STUDENT_REASONING_COLUMN="student_reasoning">>df=annotator.get_student_reasoning(df=df,speaker_column=SPEAKER_COLUMN,text_column=TEXT_COLUMN,output_column=STUDENT_REASONING_COLUMN,# Since this model is only trained on _student_ utterances,# we can explicitly pass in the speaker names associated to students.# It will only annotate utterances from these speakers.speaker_value=known_replacement_names,)# The uptake annotations will be populated in this column.>>UPTAKE_COLUMN="uptake">>df=annotator.get_uptake(df=df,speaker_column=SPEAKER_COLUMN,text_column=TEXT_COLUMN,output_column=UPTAKE_COLUMN,# Conversation uptake is about how much the teacher builds on what the students say.# So, we want to specify the first speaker to be the students.speaker1=known_replacement_names,# And the second speaker to be the teacher.speaker2='T')>>df.head(20)With these annotations, we can now do some analysis on our data. We can save the annotated data to a file for later use.>>df.to_csv("annotated_data.csv",index=False)AnalysisAnalyzing your data withedu-convokitcan help you understand the language used by your students and educators.
Analysis on education data can happen in many ways.🔍 It can happen qualitatively where you look at the data; for example, we annotated the data for student_reasoning and you might be interested in looking at the specific instances of student reasoning.📊 It can also happen quantitatively where you look at the data in aggregate; for example, you might be interested in the average amount of time the student and educator talk.💬 You might be interested in lexically analyzing the data; for example, you might be interested in what words the student uses the most compared to the educator.📈 You might be interested in temporally analyzing the data; for example, you might be interested in the amount of time the student and educator talk over the course of their interaction.🤖 You might also be interested in usingGPTto analyze the data; for example, you might want GPT4 to summarize the transcript.Here's an example of usinganalyzerfor each for these analysis types. We'll be using the same dataframe from the previous example; sodfwill be the dataframe with annotations.🔍 Qualitative AnalysisLet’s start by looking at the data qualitatively. We will use theQualitativeAnalyzerto look at the data.
Ever wondered whether there was a quick way to …Look at examples of student reasoning?Look at examples of high conversational uptake by the educator?This is where theQualitativeAnalyzercomes in handy!Let's look at some examples of student reasoning.>>fromedu_convokit.analyzersimportQualitativeAnalyzer,>>qual_analyzer=QualitativeAnalyzer()>>qual_analyzer.print_examples(df=df,speaker_column=SPEAKER_COLUMN,text_column=TEXT_COLUMN,# We want to look at examples for the reasoning featurefeature_column=STUDENT_REASONING_COLUMN,# We want to look at positive examples of reasoningfeature_value=1.0,# Let's look at 3 examplesmax_num_examples=3)student_reasoning:1.0>>[STUDENT_1]:onehalfbyonesixth.Causeifyouputsixonesuptoawholestudent_reasoning:1.0>>[STUDENT_0]:Thiswouldbeonethird,andthisisonehalfofdarkgreen,andthenitwouldbebiggerbyonesixth,becausestudent_reasoning:1.0>>[STUDENT_1]:Imeanit's bigger than one tenth, I mean twelfth, one twelfth, one  twelfth.🎉 Great! This is a quick way to look at examples of student reasoning.If you’re curious in looking at the preceding and succeeding utterances (for context), you can easily do that by specifying the number of lines:>>qual_analyzer.print_examples(df=df,speaker_column=SPEAKER_COLUMN,text_column=TEXT_COLUMN,feature_column=STUDENT_REASONING_COLUMN,feature_value=1.0,max_num_examples=3,# We want to look at the previous 2 lines of contextshow_k_previous_lines=2,# We also want to look at the next 1 line of contextshow_k_next_lines=1)student_reasoning:1.0[STUDENT_1]and[STUDENT_0]:isbiggerthanT:Youbothagree?>>[STUDENT_1]:onehalfbyonesixth.Causeifyouputsixonesuptoawhole[STUDENT_0]:darkgreenstudent_reasoning:1.0T:Ok[STUDENT_1]:Sothere’ssixsixths>>[STUDENT_0]:Thiswouldbeonethird,andthisisonehalfofdarkgreen,andthenitwouldbebiggerbyonesixth,becauseT:Doyoubothagreewiththat?student_reasoning:1.0[STUDENT_1]:Yeah,andyoutakethetwodarkgreenrods,thosearethehalves...Andyoutaketwothirds,andputituptoit,andyoutake...twosixths,it's bigger than  two sixths. And  in this one, it you take this ...T2:Canwegobacktothatoneagain?>>[STUDENT_1]:Imeanit's bigger than one tenth, I mean twelfth, one twelfth, one  twelfth.T2:Howdoesthatwork?I'm confused about that. I’m confused about  the little white rods, I am following you right up to that point.🎉 Awesome! Another handy way to use QualitativeAnalyzer is to look at both positive and negative examples of student reasoning. You can do that by omitting the feature values specification:>>qual_analyzer.print_examples(df=df,speaker_column=SPEAKER_COLUMN,text_column=TEXT_COLUMN,feature_column=STUDENT_REASONING_COLUMN,max_num_examples=3,show_k_previous_lines=2,show_k_next_lines=1,# feature_value="1.0", Omitted!)student_reasoning:0.0T:I'm wondering which is bigger, one half or two thirds. Now  before you model it you might think in your head, before you begin  to model it what you is bigger and if so, if one is bigger, by how  much. Why don’t you work with your partner and see what you can  do.>>[STUDENT_0]:Trythepurples.Getthreepurples.Itdoesn’twork,trythegreens[STUDENT_1]:Whatwasit?Twothirds?student_reasoning:0.0[STUDENT_0]:Trythepurples.Getthreepurples.Itdoesn’twork,trythegreens[STUDENT_1]:Whatwasit?Twothirds?>>[STUDENT_0]:Itwouldbelikebrownorsomethinglikethat.[STUDENT_1]:Okstudent_reasoning:0.0[STUDENT_0]:Itwouldbelikebrownorsomethinglikethat.[STUDENT_1]:Ok>>[STUDENT_0]:We’renotdoingtheonethird,we’redoingtwothirds.Thatisonethird[STUDENT_1]:Firstwe’vegottofindoutwhatathirdofitis.What’sathirdofanorange?student_reasoning:1.0[STUDENT_1]and[STUDENT_0]:isbiggerthanT:Youbothagree?>>[STUDENT_1]:onehalfbyonesixth.Causeifyouputsixonesuptoawhole[STUDENT_0]:darkgreenstudent_reasoning:1.0T:Ok[STUDENT_1]:Sothere’ssixsixths>>[STUDENT_0]:Thiswouldbeonethird,andthisisonehalfofdarkgreen,andthenitwouldbebiggerbyonesixth,becauseT:Doyoubothagreewiththat?student_reasoning:1.0[STUDENT_1]:Yeah,andyoutakethetwodarkgreenrods,thosearethehalves...Andyoutaketwothirds,andputituptoit,andyoutake...twosixths,it's bigger than  two sixths. And  in this one, it you take this ...T2:Canwegobacktothatoneagain?>>[STUDENT_1]:Imeanit's bigger than one tenth, I mean twelfth, one twelfth, one  twelfth.T2:Howdoesthatwork?I'm confused about that. I’m confused about  the little white rods, I am following you right up to that point.Looking at examples of other annotations is similar---just specify the feature column and feature value.📊 Quantitative AnalysisLet’s start by looking at the data quantitatively. We will use theQuantitativeAnalyzerto look at the data.Ever wondered whether there was a quick way to report aggregate statistics on:Talk time?Student reasoning?Conversational uptake?This is where the QuantitativeAnalyzer comes in handy!Let’s say we want to understand the talk time percentage split between the student and educator. We might want to know the statistic and plot the data. Here’s how we can do that:>>fromedu_convokit.analyzersimportQuantitativeAnalyzer>>analyzer=QuantitativeAnalyzer()# Create speaker mapping to A, B, C, D>>label_mapping={speaker:chr(ord('A')+i)fori,speakerinenumerate(df[SPEAKER_COLUMN].unique())}>>analyzer.plot_statistics(# Everything else is the samefeature_column=TALK_TIME_COLUMN,df=df,speaker_column=SPEAKER_COLUMN,value_as="prop",# We want to use the mapping we createdlabel_mapping=label_mapping)The other statistics are similar to the example above for talk time. For conciseness, we will omit them here; but the only thing you need to do is change the feature_column argument to the feature you want to analyze.💬 Lexical AnalysisA lexical analysis is an analysis on the words used in the data. This is useful for understanding the low-level language (i.e., word usage) used by the student and educator. While a lexical analysis may be too low-level for capturing e.g., the meaning of the discourse, it can be a useful first step in capturing language trends in the data.We will give a simple demonstration of two features of theLexicalAnalyzer:Word Frequency: We will look at the most frequent words used by each speaker.Log-Odds: We will look at the log-odds of words used by each speaker (i.e., which words are more likely to be used by the student vs. the educator).>>fromedu_convokit.analyzersimportLexicalAnalyzer>>analyzer=LexicalAnalyzer()>>analyzer.print_word_frequency(df=df,text_column=TEXT_COLUMN,speaker_column=SPEAKER_COLUMN,# We want to look at the top 5 wordstopk=5,# We want to format the text e.g., remove punctuation and stopwords (https://en.wikipedia.org/wiki/Stop_word)run_text_formatting=True)TopWordsBySpeakerTworks:6one:5bigger:4models:4write:3[STUDENT_0]one:10would:5third:4yeah:3bigger:3[STUDENT_1]one:16two:13take:9put:8yeah:8[STUDENT_2]one:1half:1[STUDENT_1]and[STUDENT_0]well:1bigger:1one:1T2one:9two:8interesting:5okay:4twelfths:4💡 We can see that there’s a lot of use of numbers and fractions (“one”, “third”), in addition to comparison language (“bigger”).If you want to see the most frequent words overall, you can omit the speaker_column argument:>>analyzer.print_word_frequency(df=df,text_column=TEXT_COLUMN,# Bye! We don't care about the speaker anymore# speaker_column=SPEAKER_COLUMN,topk=5,run_text_formatting=True)one:42two:25bigger:15yeah:12sixth:11Let’s now move onto the log-odds analysis.💡 Why a log-odds analysis?Going beyond just counting frequent words in the student and teacher’s utterances, we might be interested in the chances of a word occurring in the student’s text over it occurring in the teacher’s text.This gets us to a log-odds analysis on the words. For more information on log-odds analysis, please refer to this paper which applied the same analysis to study language use in political speeches.In order to run a log-odds analysis, we need to specify two groups of texts we want to compare to each other. In this case, we are interested in comparing the student’s text to the teacher’s text. So let’s split our original dataframe into these two groups and pass them into the log-odds analysis.>>speakers=df[SPEAKER_COLUMN].unique()# Student speakers are ones that contain STUDENT in their name>>student_speakers=[speakerforspeakerinspeakersif"STUDENT"inspeaker]# Teacher speakers are all the other speakers>>teacher_speakers=[speakerforspeakerinspeakersifspeakernotinstudent_speakers]# Now let's split the data frame into two data frames: one for student speakers and one for teacher speakers>>student_df=df[df[SPEAKER_COLUMN].isin(student_speakers)]>>teacher_df=df[df[SPEAKER_COLUMN].isin(teacher_speakers)]# We can now run the analyzer:>>analyzer.print_log_odds(df1=student_df,df2=teacher_df,text_column1=TEXT_COLUMN,text_column2=TEXT_COLUMN,# We want to look at the top 5 wordstopk=5,# We still want to format the textrun_text_formatting=True)TopwordsforGroup1put:1.5082544501223418yeah:1.4243179151183505third:1.323889968245073take:1.2333974083596422green:1.1289734037070032TopwordsforGroup2works:-1.5890822958913073models:-1.5890822958913073interesting:-1.447190205063502see:-1.291347559145683okay:-1.291347559145683🎉 Awesome, this shows us both the words and odds of the words used by the student (Group 1) and educator (Group 2).We can also plot this with a similar function call:analyzer.plot_log_odds(df1=student_df,df2=teacher_df,text_column1=TEXT_COLUMN,text_column2=TEXT_COLUMN,topk=5,run_text_formatting=True,# We can pass plot labels for group 1 and 2group1_name="Student",group2_name="Teacher")📈 Temporal AnalysisA temporal analysis is an analysis on the features over time. We define time as the time over the course of the transcript.Let’s see how we can look at the talk time ratio (which we summarized before quantitatively) over time. This setup will look similar to the quantitative analysis however the key difference is that we will specify a num_bins argument to specify the number of bins we want to split the transcript into.>>fromedu_convokit.analyzersimportTemporalAnalyzer>>analyzer=TemporalAnalyzer()# We'll use the same label mapping as before to abbreviate the speaker names>>print(f"Label mapping:{label_mapping}")Labelmapping:{'T':'A','[STUDENT_0]':'B','[STUDENT_1]':'C','[STUDENT_2]':'D','[STUDENT_1] and [STUDENT_0]':'E','T 2':'F'}>>analyzer.plot_temporal_statistics(feature_column=TALK_TIME_COLUMN,dfs=df,speaker_column=SPEAKER_COLUMN,# We want to see the proportion of talk timevalue_as="prop",# We will split the session into 5 binsnum_bins=5,# We want to use the mapping we createdlabel_mapping=label_mapping)🎉 Plotting the temporal data was straightforward!💡 There are some interesting observations, for example:Speaker B ([STUDENT_0]) decreases their talk time over time whereasSpeaker C (the other student, [STUDENT_1]) has fluctuating talk time over time.Plotting the other statistics is similar to the example above for talk time. For conciseness, we will omit them here; but the only thing you need to do is change the feature_column argument to the feature you want to analyze.🤖 GPT AnalysisA GPT analysis is an analysis using GPT (e.g., ChatGPT or GPT4) to analyze the data.
This is useful for understanding the high-level language (i.e., meaning) used by the student and educator.
While a GPT analysis may be too high-level for capturing e.g., the specific words used by the student and educator, it can be a useful first step in capturing meaning in the data.Let’s see how we can use GPT to summarize the transcript. We will use theGPTConversationAnalyzerto do this. There are other prompts you can use for GPT inthis directory.You will first need to set your OpenAI API key as an environment variable.>>importos# Remember to never share your API key with anyone!>>os.environ["OPENAI_API_KEY"]="YOUR_API_KEY"We can now try to summarize the transcript.>>fromedu_convokit.analyzersimportGPTConversationAnalyzer>>analyzer=GPTConversationAnalyzer()>>prompt=analyzer.preview_prompt(df=df,# Using the summarize prompt under prompts/conversationprompt_name="summarize",text_column=TEXT_COLUMN,speaker_column=SPEAKER_COLUMN,model="gpt-4",format_template="{speaker}:{text}",)>>print(prompt)Considerthefollowingtranscriptofaconversationbetweenateacherandastudent.Transcript:T:I'm wondering which is bigger, one half or two thirds. Now  before you model it you might think in your head, before you begin  to model it what you is bigger and if so, if one is bigger, by how  much. Why don’t you work with your partner and see what you can  do.[STUDENT_0]:Trythepurples.Getthreepurples.Itdoesn’twork,trythegreens[STUDENT_1]:Whatwasit?Twothirds?[STUDENT_0]:Itwouldbelikebrownorsomethinglikethat.[STUDENT_1]:Ok[STUDENT_0]:We’renotdoingtheonethird,we’redoingtwothirds.Thatisonethird[STUDENT_1]:Firstwe’vegottofindoutwhatathirdofitis.What’sathirdofanorange?[STUDENT_0]:Onethird?[STUDENT_1]:What’sthirdofanorange?Let’sstartadifferentmodel.Thegreen.Thegreen,halfofitisthelightgreen[STUDENT_0]:Alright,yeah,Iwasthinkingofthatwaybefore[STUDENT_1]:Andyoucantakethetakethered,andthelightgreen,andputituptoit,it’s,sheasked,isonehalfbiggerthan,whatdidsheask?Whatdidsheask?[STUDENT_0]:Sheasked,whichisbigger,onehalfortwothirds?...[STUDENT_1]:Uhhmm.T2:Andoverhereoneofthewhitesyousayisonesixth?[STUDENT_1]:Yeah.T2:Oh,that’sinteresting,twodifferentmodels.Okay.Pleasesummarizetheconversationinafewsentences.Summary:💡 We can see that the prompt is set up to summarize the transcript. If you are comfortable with the prompt, you can now prompt the model on that prompt withrun_prompt>>prompt,response=analyzer.run_prompt(df=df,prompt_name="summarize",text_column=TEXT_COLUMN,speaker_column=SPEAKER_COLUMN,model="gpt-4",format_template="{speaker}:{text}",)>>print(response)Theteacheraskedthestudentstodeterminewhichislarger,onehalfortwothirds,andbyhowmuch.Thestudentsuseddifferentcoloredrodstorepresentfractionsandconcludedthattwothirdsislargerthanonehalfbyonesixth.Theythentestedtheirtheorywithdifferentmodels,usingdifferentrodstorepresentdifferentfractions,andfoundthattheirconclusionheldtrue.Theteacherencouragedthemtowriteuptheirfindingsandexplainthemtotheclass.There are many other prompts that you can perform a GPT analysis on. For more information, please see our prompts databasehere.Papers that have used theedu-convokitPlease findherea list of papers that have used theedu-convokit.Future FeaturesAdd GPT feature annotationCitationIf you use theedu-convokitin your research, please cite the following paper:@article{wang2024educonvokit,
  title={Edu-ConvoKit: An Open-Source Framework for Education Conversation Data},
  author={Wang, Rose E and Demszky, Dorottya},
  year={2024}
}If you would like to be added to the list of papers that have used theedu-convokit, please make a pull request or contact Rose E. Wang atrewang@cs.stanford.edu.ContactIf you have any questions, please contact Rose E. Wang atrewang@cs.stanford.edu.