Text embedding models yield detailed conceptual knowledge maps derived from short multiple-choice quizzes

Domain: biology
Published: 2026-03-24
Editorial outcome: major_revision

Abstract
Real-world conceptual knowledge is complex, multifaceted, and substantially over-simplified in most laboratory studies. Here we develop a mathematical framework, based on natural language processing models, for tracking and characterizing the acquisition of real-world conceptual knowledge. Our approach embeds each concept in a high-dimensional representation space where nearby coordinates reflect similar or related concepts. We test our approach using behavioral data from participants who answered small sets of multiple-choice quiz questions interleaved between watching two Khan Academy course videos. We apply our framework to the videos’ transcripts and the text of the quiz questions to quantify the content of each moment of video and each quiz question. We use these embeddings, along with participants’ quiz responses, to track how the learners’ knowledge changed after watching each video and predict their success on individual quiz questions. Our findings show how a small set of quiz questions may be used to obtain rich and meaningful insights into what each learner knows, and how their knowledge changes over time as they learn.

Introduction
Suppose that a teacher had access to a complete, tangible map of everything a student knows. Defining what such a map might even look like, let alone how it might be constructed or filled in, is itself a non-trivial problem. But if a teacher were to gain access to such a map, how might it change their ability to teach that student? Perhaps they might start by checking how well the student knows the to-be-learned information already, or how much they know about related concepts. For some students, they could potentially optimize their teaching efforts to maximize efficiency by focusing primarily on not-yet-known content. For other students (or other content areas), it might be more effective to optimize for direct connections between already known content and new material. Observing how the student’s knowledge changed over time, in response to their teaching, could also help to guide the teacher toward the most effective strategy for that individual student.

A common approach to assessing a student’s knowledge is to present them with a set of quiz questions, calculate the proportion they answer correctly, and provide them with feedback in the form of a simple numeric or letter grade. While such a grade can provide some indication of whether the student has mastered the to-be-learned material, any univariate measure of performance on a complex task sacrifices certain relevant information, risks conflating underlying factors, and so on. For example, consider the relative utility of the theoretical map described above that characterizes a student’s knowledge in detail, versus a single annotation saying that the student answered 85% of their quiz questions correctly, or that they received a ‘B’. Here we show that the same quiz data required to compute proportion-correct scores or letter grades can instead be used to obtain far more detailed insights into what a student knew at the time they took the quiz.

Designing and building procedures and tools for mapping out knowledge touches on deep questions about what it means to learn. For example, how do we acquire conceptual knowledge? Memorizing course lectures or textbook chapters by rote can lead to the superficial appearance of understanding the underlying content, but achieving true conceptual understanding seems to require something deeper and richer. Does conceptual understanding entail connecting newly acquired information to the scaffolding of one’s existing knowledge or experience1–6? Or weaving a lecture’s atomic elements (e.g., its component words) into a structured network that describes how those individual elements are related7,8? Conceptual understanding could also involve building a mental model that transcends the meanings of those individual atomic elements by reflecting the deeper meaning underlying the gestalt whole9–12.

The difference between “understanding” and “memorizing,” as framed by researchers in education, cognitive psychology, and cognitive neuroscience (e.g., 10,11,13–15), has profound analogs in the fields of natural language processing and natural language understanding. For example, considering the raw contents of a document (e.g., its constituent symbols, letters, and words) might provide some clues as to what the document is about, just as memorizing a passage might provide some ability to answer simple questions about it. However, text embedding models (e.g., 16–23) also attempt to capture the deeper meaning underlying those atomic elements. These models consider not only the co-occurrences of those elements within and across documents, but (in many cases) also patterns in how those elements appear across different scales (e.g., sentences, paragraphs, chapters, etc.), their temporal and grammatical properties, and other high-level characteristics of how they are used24,25. To be clear, this is not to say that text embedding models themselves are capable of “understanding” deep conceptual meaning in any traditional sense. But rather, their ability to capture the underlying structure of text documents beyond their surface-level contents provides a computational framework through which those documents’ deeper conceptual meanings may be quantified, explored, and understood. According to these models, the deep conceptual meaning of a document may be captured by a feature vector in a high-dimensional representation space, wherein nearby vectors reflect conceptually related documents. A model that succeeds at capturing an analogue of “understanding” is able to assign nearby feature vectors to two conceptually related documents even when the specific words contained in those documents have limited overlap. In this way, “concepts” are defined implicitly by the model’s geometry (e.g., how the embedding coordinate of a given word or document relates to the coordinates of other text embeddings26).

Given these insights, what form might a representation of the sum total of a person’s knowledge take? First, we might require a means of systematically describing or representing (at least some subset of) the nearly infinite set of possible things a person could know. Second, we might want to account for potential associations between different concepts. For example, the concepts of “fish” and “water” might be associated in the sense that fish live in water. Third, knowledge may have a critical dependency structure, such that knowing about a particular concept might require first knowing about a set of other concepts. For example, understanding the concept of a fish swimming in water first requires understanding what fish and water are. Fourth, as we learn, our current state of knowledge should change accordingly. Learning new concepts should both update our characterizations of what is known and also unlock any now-satisfied dependencies of those newly learned concepts so that they are considered available for future learning.

Here we develop a framework for modeling how conceptual knowledge is acquired during learning. The central idea behind our framework is to use text embedding models to define the coordinate systems of two maps: a “knowledge map” that describes the extent to which each concept is currently known, and a “learning map” that describes changes in knowledge over time. Each location on these maps represents a single concept, and the maps’ geometries are defined such that related concepts are located nearby in space. We use this framework to analyze and interpret behavioral data collected from an experiment that had participants answer sets of multiple-choice questions about a series of recorded course lectures.

Our primary research goal is to advance our understanding of what it means to acquire deep, real-world conceptual knowledge. Traditional laboratory approaches to studying learning and memory (e.g., list-learning studies) often draw little distinction between memorization and understanding. Instead, these studies typically focus on whether information is effectively encoded or retrieved, rather than whether the information is understood. Approaches to studying conceptual learning, such as category learning experiments, can begin to investigate the distinction between memorization and understanding, often by training participants to distinguish arbitrary or random features in otherwise meaningless categorized stimuli27–32. However, the objective of real-world training, or learning from life experiences more generally, is often to develop new knowledge that may be applied in useful ways in the future. In this sense, the gap between modern learning theories and modern pedagogical approaches that inform classroom learning strategies is enormous: most of our theories about how people learn are inspired by experimental paradigms and models that have only peripheral relevance to the kinds of learning that students and teachers actually seek10,15. To help bridge this gap, our study uses course materials from real online courses to inform, fit, and test models of real-world conceptual learning. We show that these models recover meaningful relationships between concepts presented during course lectures and tested by assessments, and that these relationships can be leveraged to predict students’ success on individual quiz questions. We also provide a demonstration of how our models can be used to construct maps of what students know, and how their knowledge changes with training. In addition to helping to visually capture knowledge (and changes in knowledge), we hope that such maps might lead to real-world tools for improving how we educate. Taken together, our work shows that existing course materials and evaluative tools like short multiple-choice quizzes may be leveraged to gain highly detailed insights into what students know and how they learn.

Results
At its core, our main modeling approach is based around a simple assumption that we sought to test empirically: all else being equal, knowledge about a given concept is predictive of knowledge about similar or related concepts. From a geometric perspective, this assumption implies that knowledge is fundamentally “smooth.” In other words, as one moves through a space representing an individual’s knowledge (where similar concepts occupy nearby coordinates), their level of knowledge should change relatively gradually. To begin to test this smoothness assumption, we sought to track participants’ knowledge and how it changed over time in response to training. Two overarching goals guide our approach. First, we want to gain detailed insights into what learners know at different points in their training. For example, rather than simply reporting on the proportions of questions participants answer correctly (i.e., their overall performance), we seek estimates of their knowledge about a variety of specific concepts. Second, we want our approach to be potentially scalable to large numbers of diverse concepts, courses, and students. This requires that the conceptual content of interest be discovered automatically, rather than relying on manually produced ratings or labels.

We asked participants in our study to complete brief multiple-choice quizzes before, between, and after watching two lecture videos from the Khan Academy33 platform (Fig. 1). The first lecture video, entitled Four Fundamental Forces, discussed the four fundamental forces in physics: gravity, strong and weak interactions, and electromagnetism. The second, entitled Birth of Stars, provided an overview of our current understanding of how stars form. We selected these particular lectures to satisfy three general criteria. First, we wanted both lectures to be accessible to a broad audience (i.e., with minimal prerequisite knowledge) so as to limit the impact of prior training on participants’ abilities to learn from the lectures. To this end, we selected two introductory videos that were intended to be viewed at the start of students’ training in their respective content areas. Second, we wanted the two lectures to have some related content so that we could test our approach’s ability to distinguish similar conceptual content. To this end, we chose two videos from the same Khan Academy course domain, “Cosmology and Astronomy.” Third, we sought to minimize dependencies and specific overlap between the videos. For example, we did not want participants’ abilities to understand one video to (directly) influence their abilities to understand the other. To satisfy this last criterion, we chose videos from two different lecture series (Lectures 1 and 2 were from the “Scale of the Universe” and “Stars, Black Holes, and Galaxies” series, respectively).

We also wrote a set of multiple-choice quiz questions that we hoped would enable us to evaluate participants’ knowledge about each individual lecture, along with related knowledge about physics concepts not specifically presented in either video (see Supplementary Table 1 for the full list of questions in our stimulus pool). Participants answered questions randomly drawn from each content area (Lecture 1, Lecture 2, and general physics knowledge) on each of the three quizzes. Quiz 1 was intended to assess participants’ baseline knowledge before training, Quiz 2 assessed knowledge after watching the Four Fundamental Forces video (i.e., Lecture 1), and Quiz 3 assessed knowledge after watching the Birth of Stars video (i.e., Lecture 2).

To study in detail how participants’ conceptual knowledge changed over the course of the experiment, we first sought to model the conceptual content presented to them at each moment throughout each of the two lectures. We adapted an approach we developed in prior work34 to identify the latent themes in the lectures using a topic model18. Briefly, topic models take as input a collection of text documents, and learn a set of “topics” (i.e., latent themes) from their contents. Once fit, a topic model can be used to transform arbitrary (potentially new) documents into sets of “topic proportions” describing the weighted blend of learned topics reflected in their texts. We parsed automatically generated transcripts of the two lectures into overlapping sliding windows, where each window contained the text of the lecture transcript from a particular timespan. We treated the set of text snippets (across all of these windows) as documents to fit the model (Fig. 2A; see Constructing text embeddings of multiple lectures and questions). Transforming the text from every sliding window with the model yielded a number-of-windows by number-of-topics (15) topic-proportions matrix describing the unique mixture of broad themes from both lectures reflected in each window’s text. Each window’s “topic vector” (i.e., row of the topic-proportions matrix) is analogous to a coordinate in a 15-dimensional space whose axes are topics discovered by the model. Within this space, each lecture’s sequence of topic vectors (i.e., corresponding to its transcript’s overlapping text snippets across sliding windows) forms a trajectory that captures how its conceptual content unfolds over time (Fig. 2B). We resampled these trajectories to a resolution of one topic vector for each second of video (i.e., 1 Hz).

We hypothesized that a topic model trained on transcripts of the two lectures should also capture the conceptual knowledge probed by each quiz question. If indeed the topic model could capture information about the deeper conceptual content of the lectures (i.e., beyond surface-level details such as particular word choices), then we should be able to recover a correspondence between each lecture and questions about each lecture. Importantly, such a correspondence could not arise solely from superficial text matching between lecture transcripts and questions, since the lectures and questions often used different words (Supplementary Fig. 11) and phrasings. Simply comparing the average topic weights from each lecture and question set (averaging across time and questions, respectively) reveals a striking correspondence (Supplementary Fig. 2). Specifically, the average topic weights from Lecture 1 are strongly correlated with the average topic weights from questions about Lecture 1 (r(13) = 0.809, p < 0.001, 95% confidence interval (CI) = [0.633, 0.962]), and the average topic weights from Lecture 2 are strongly correlated with the average topic weights from questions about Lecture 2 (r(13) = 0.728, p = 0.002, 95% CI = [0.456, 0.920]). At the same time, the average topic weights from the two lectures are negatively correlated with the average topic weights from their non-matching question sets (Lecture 1 video vs. Lecture 2 questions: r(13) = −0.547, p = 0.035, 95% CI = [−0.812, −0.231]; Lecture 2 video vs. Lecture 1 questions: r(13) = −0.612, p = 0.015, 95% CI = [−0.874, −0.281]), indicating that the topic model also exhibits some degree of specificity. The full set of pairwise comparisons between average topic weights for the lectures and question sets is reported in Supplementary Fig. 2.

Another, more sensitive, way of summarizing the conceptual content of the lectures and questions is to look at variability in how topics are weighted over time and across different questions (Fig. 3). Intuitively, the variability in the expression of a given topic relates to how much “information”35 the lecture (or question set) reflects about that topic. For example, suppose a given topic is weighted on heavily throughout a lecture. That topic might be characteristic of some aspect or property of the lecture overall (conceptual or otherwise), but unless the topic’s weights change in meaningful ways over time, it would be a poor indicator of any specific conceptual content in the lecture. We therefore also compared the variances in topic weights (over time and across questions) between the lectures and questions. The variability in topic expression was similar for the Lecture 1 video and questions (r(13) = 0.824, p < 0.001, 95% CI = [0.696, 0.973]), and for the Lecture 2 video and questions (r(13) = 0.801, p < 0.001, 95% CI = [0.539, 0.958]). Simultaneously, as reported in Fig. 3B, the variabilities in topic expression across different videos and lecture-specific questions (i.e., Lecture 1 video vs. Lecture 2 questions; Lecture 2 video vs. Lecture 1 questions) were negatively correlated, and neither video’s topic variability was reliably correlated with the topic variability across general physics knowledge questions. Taken together, the analyses reported in Fig. 3 and Supplementary Fig. 2 indicate that a topic model fit to the videos’ transcripts can also reveal correspondences (at a coarse scale) between the lectures and questions.

An individual lecture may be organized around a single broad theme at a coarse scale, but at a finer scale, each moment of a lecture typically covers a narrower range of content. Given the correspondence we found between the variabilities in topic expression across moments of each lecture and questions from its corresponding set (Fig. 3), we wondered whether the text embedding model might additionally capture these conceptual relationships at a finer scale. For example, if a particular question asks about the content from one small part of a lecture, we wondered whether the text embeddings could be used to automatically identify the matching moment(s) in the lecture. To explore this, we computed the correlation between each question’s topic weights and the topic weights for each second of its corresponding lecture, and found that each question appeared to be temporally specific (Fig. 4). In particular, most questions’ topic vectors were maximally correlated with a well-defined (and relatively narrow) range of timepoints from their corresponding lectures, outside of which the correlations fell off sharply (Supplementary Figs. 3, 4). We also qualitatively examined the best-matching intervals for each question by comparing the questions’ text to the transcribed text from the most-correlated parts of the lectures (Supplementary Table 3). Despite that the questions were excluded from the text embedding model’s training set, in general we found (through manual inspection) a close correspondence between the conceptual content that each question probed and the content covered by the best-matching moments of the lectures. Two representative examples are shown at the bottom of Fig. 4.

The ability to quantify how much each question is asking about the content from each moment of the lectures could enable more detailed insights into participants’ knowledge. Traditional approaches to estimating how much a student knows about the content of a given lecture entail administering some form of assessment (e.g., a quiz) and computing the proportion of questions the student answered correctly. But if two students receive identical scores on such an assessment, might our modeling framework help us to gain more nuanced insights into the specific content that each student has mastered (or failed to master)? For example, a student who misses three questions that were all about the same concept (e.g., concept A) will have gotten the same proportion of questions correct as another student who missed three questions about three different concepts (e.g., A, B, and C). But if we wanted to help these two students fill in the gaps in their understandings, we might do well to focus specifically on concept A for the first student, but to also add in materials pertaining to concepts B and C for the second student. In other words, raw proportion-correct measures may capture how much a student knows, but not what they know. We wondered whether our modeling framework might enable us to (formally and automatically) infer participants’ knowledge at the scale of individual concepts (e.g., as captured by a single moment of a lecture).

We developed a simple formula (Eq. (1)) for using a participant’s responses to a small set of multiple-choice questions to estimate how much that participant knows about the concept reflected by any arbitrary coordinate x in text embedding space (e.g., the content reflected by any moment in a lecture they had watched; see Estimating dynamic knowledge traces). Essentially, the estimated knowledge at coordinate x is given by the weighted proportion of quiz questions the participant answered correctly, where the weights reflect how much each question is “about” the content at x. When we apply this approach to estimate the participant’s knowledge about the content presented in each moment of each lecture, we can obtain a detailed time course describing how much knowledge that participant has about the content presented at any part of the lecture. As shown in Fig. 5A, C, we can apply this approach separately for the questions from each quiz participants took throughout the experiment. From just a few questions per quiz (see Estimating dynamic knowledge traces), we obtain a high-resolution snapshot (at the time each quiz was taken) of what participants knew about any moment’s content, from either of the two lectures they watched (comprising a total of 1100 samples across the two lectures).

While the time courses in Fig. 5A, C provide detailed estimates about participants’ knowledge, these estimates are of course only useful to the extent that they accurately reflect what participants actually know. As one sanity check, we anticipated that the knowledge estimates should reflect a content-specific boost in participants’ knowledge after watching each lecture. In other words, if participants learn about each lecture’s content upon watching it, the knowledge estimates should capture that. After watching the Four Fundamental Forces lecture, participants should exhibit more knowledge for the content of that lecture than they had before, and that knowledge should persist for the remainder of the experiment. Specifically, knowledge about that lecture’s content should be relatively low when estimated using Quiz 1 responses, but should increase when estimated using Quiz 2 or 3 responses (Fig. 5B). Indeed, we found that participants’ estimated knowledge about the content of Four Fundamental Forces was substantially higher on Quiz 2 versus Quiz 1 (t(49) = 8.764, p < 0.001) and on Quiz 3 versus Quiz 1 (t(49) = 10.519, p < 0.001). We found no reliable differences in estimated knowledge about that lecture’s content on Quiz 2 versus 3 (t(49) = 0.160, p = 0.874). Similarly, we hypothesized (and subsequently confirmed) that participants should show greater estimated knowledge about the content of the Birth of Stars lecture after (versus before) watching it (Fig. 5D). Specifically, since participants watched that lecture after taking Quiz 2 (but before Quiz 3), we hypothesized that their knowledge estimates should be relatively low on Quizzes 1 and 2, but should show a boost on Quiz 3. Consistent with this prediction, we found no reliable differences in estimated knowledge about the Birth of Stars lecture content on Quiz 1 versus 2 (t(49) = 1.013, p = 0.316), but estimated knowledge was substantially higher on Quiz 3 versus 2 (t(49) = 10.561, p < 0.001) and Quiz 3 versus 1 (t(49) = 8.969, p < 0.001).

If we are able to accurately estimate a participant’s knowledge about the content tested by a given question, our estimates of their knowledge should carry some predictive information about whether they are likely to answer that question correctly or incorrectly. We developed a statistical approach to test this claim. For each quiz question a participant answered, in turn, we used Eq. (1) to estimate their knowledge at that question’s embedding-space coordinate based on other questions that participant answered on the same quiz. We repeated this for all participants, and for each of the three quizzes. Then, separately for each quiz, we fit a generalized linear mixed model (GLMM) with a logistic link function to explain the probability of correctly answering a question as a function of estimated knowledge at its embedding coordinate, while accounting for varied effects of individual participants and questions (see Generalized linear mixed models). To assess the predictive value of the knowledge estimates, we compared each GLMM to an analogous (i.e., nested) null model that assumed these estimates carried no predictive information using parametric bootstrap likelihood-ratio tests.

We carried out three different versions of the analyses described above, wherein we considered different sources of information in our estimates of participants’ knowledge for each quiz question. First, we estimated knowledge at each held-out question’s embedding coordinate using all other questions answered by the same participant on the same quiz (“All questions”; Fig. 6, top row). This test was intended to assess the overall predictive power of our approach. Second, we estimated knowledge for each question about a given lecture using only the other questions (from the same participant and quiz) about that same lecture (“Within-lecture”; Fig. 6, middle rows). This test was intended to assess the specificity of our approach by asking whether our predictions could distinguish between questions about different content covered by the same lecture. Third, we estimated knowledge for each question about one lecture using only the questions (from the same participant and quiz) about the other lecture (“Across-lecture”; Fig. 6, bottom rows). This test was intended to assess the generalizability of our approach by asking whether our predictions could extend across the content areas of the two lectures. When estimating participants’ knowledge, we used a rebalancing procedure to ensure that (for a given participant and quiz) their knowledge estimates for correctly and incorrectly answered questions were computed from the same underlying proportion of correctly answered questions (see Generalized linear mixed models).

When we fit a GLMM to estimates of participants’ knowledge for each Quiz 1 question based on all other Quiz 1 questions, we found that higher estimated knowledge for a given question predicted a greater likelihood of answering it correctly (odds ratio (OR) = 8.126, 95% CI = [3.116, 20.123], likelihood-ratio test statistic (λLR) = 17.002, p < 0.001). This relationship held when we repeated this analysis for Quiz 2 (OR = 14.902, 95% CI = [4.976, 39.807], λLR = 25.408, p < 0.001) and again for Quiz 3 (OR = 37.409, 95% CI = [10.425, 107.145], λLR = 40.948, p < 0.001). Taken together, these results suggest that our knowledge estimates can reliably predict participants’ performance on individual questions when they incorporate information from all (other) quiz content.

We observed a similar set of results when we restricted our estimates of participants’ knowledge to consider only their performance on other questions about the same lecture. Specifically, for Quiz 1, participants’ knowledge of Four Fundamental Forces-related questions, estimated from their performance on other Four Fundamental Forces-related questions, was predictive of their ability to answer those questions correctly (OR = 15.934, 95% CI = [5.173, 38.005], λLR = 40.971, p < 0.001). The same was true of participants’ estimated knowledge for Birth of Stars-related questions based on their performance on other Birth of Stars-related questions (OR = 9.775, 95% CI = [2.930, 25.080], λLR = 13.924, p < 0.001). These results also held for participants’ Quiz 2 responses (Four Fundamental Forces: OR = 35.126, 95% CI = [5.113, 123.868], λLR = 32.251, p < 0.001; Birth of Stars: OR = 4.717, 95% CI = [2.021, 9.844], λLR = 16.788, p < 0.001) and partially for their Quiz 3 responses (Birth of Stars: OR = 16.902, 95% CI = [3.353, 53.265], λLR = 23.233, p < 0.001; Four Fundamental Forces: OR = 2.485, 95% CI = [0.724, 8.366], λLR = 1.984, p = 0.170). We note that the within-lecture knowledge estimates are susceptible to ceiling effects in participants’ quiz performance. For example, on Quiz 3, after viewing both lectures, no participant answered more than three Four Fundamental Forces-related questions incorrectly, and all but five participants (out of 50) answered two or fewer incorrectly. (This was the only subset of questions about either lecture, across all three quizzes, for which this was true.) Consequently, for 90% of participants, our within-lecture estimates of their knowledge for Four Fundamental Forces-related questions that they answered incorrectly leveraged information from at most a single other question they were not able to correctly answer. This hampered our ability to accurately characterize the specific (and by the time they took Quiz 3, relatively few) aspects of the lecture content these participants did not know about, and successfully distinguish them from the far more numerous aspects of the lecture content they now did know about. Taken together, these within-lecture results suggest that our knowledge estimates can reliably distinguish between questions about different content covered by a single lecture, provided there is sufficient diversity in participants’ quiz responses to extract meaningful information about both what they know and what they do not know.

Finally, we estimated participants’ knowledge for each question about each lecture using only their performance on questions (from the same quiz) about the other lecture. This is an especially stringent test of our approach. Our primary assumption in constructing our knowledge estimates is that knowledge about a given concept is similar to knowledge about other concepts that are nearby in the embedding space. However, our analyses in Fig. 3 and Supplementary Fig. 2 show that the embeddings of content from the two lectures (and of their associated quiz questions) are largely distinct from each other. Therefore, any predictive power of these across-lecture knowledge estimates must overcome large distances in the embedding space. To put this in concrete terms, this test requires predicting participants’ performance on individual, highly specific questions about the formation of stars from their responses to just five multiple-choice questions about the fundamental forces of the universe (and vice versa).

We found that, before viewing either lecture (i.e., on Quiz 1), participants’ abilities to answer Four Fundamental Forces-related questions could not be predicted from their responses to Birth of Stars-related questions (OR = 1.896, 95% CI = [0.419, 9.088], λLR = 0.712, p = 0.404), nor could their abilities to answer Birth of Stars-related questions be predicted from their responses to Four Fundamental Forces-related questions (OR = 1.522, 95% CI = [0.332, 6.835], λLR = 0.286, p = 0.611). Similarly, we found that participants’ performance on questions about either lecture could not be predicted given their responses to questions about the other lecture after viewing Four Fundamental Forces but before viewing Birth of Stars (i.e., on Quiz 2; Four Fundamental Forces questions given Birth of Stars questions: OR = 3.49, 95% CI = [0.739, 12.849], λLR = 3.266, p = 0.083; Birth of Stars questions given Four Fundamental Forces questions: OR = 2.199, 95% CI = [0.711, 5.623], λLR = 2.304, p = 0.141). Only after viewing both lectures (i.e., on Quiz 3) did these across-lecture knowledge estimates reliably predict participants’ success on individual quiz questions (Four Fundamental Forces questions given Birth of Stars questions: OR = 11.294, 95% CI = [1.375, 47.744], λLR = 10.396, p < 0.001; Birth of Stars questions given Four Fundamental Forces questions: OR = 7.302, 95% CI = [1.077, 44.879], λLR = 4.708, p = 0.038). Taken together, these results indicate that our ability to form estimates solely across different content areas is more limited than our ability to form estimates that incorporate responses to questions from both content areas (as in Fig. 6, “All questions”) or within a single content area (as in Fig. 6, “Within-lecture”). However, if participants have recently received some training on both content areas, the knowledge estimates appear to be informative even across content areas.

We speculate that these “Across-lecture” results might relate to some of our earlier work on the nature of semantic representations36. In that work, we asked whether semantic similarities could be captured through behavioral measures, even if participants’ true internal representations differed from the embeddings used to characterize their behaviors. We found that mismatches between an individual’s internal representation of a set of concepts and the representation used to characterize their behaviors can lead to underestimates of how semantically driven those behaviors are. Along similar lines, we suspect that in our current study, participants’ conceptual representations may initially differ from the representations learned by our topic model. (Although the topic model’s representations are still related to participants’ initial internal representations; otherwise we would have found that knowledge estimates derived from Quizzes 1 and 2 had no predictive power in the other tests we conducted.) After watching both lectures, however, participants’ internal representations may become more aligned with the embeddings used to estimate their knowledge (since those embeddings were trained on the lectures’ transcripts). This could help explain why the knowledge estimates derived from Quizzes 1 and 2 (before both lectures had been watched) do not reliably predict performance across content areas, whereas estimates derived from Quiz 3 do.

That the knowledge predictions derived from the text embedding space reliably distinguish between correctly and incorrectly answered held-out questions (Fig. 6) suggests that geometric relationships within this space can help explain what participants know. But how far does this explanatory power extend? For example, suppose we know that a participant correctly answered a question at embedding coordinate x. As we move farther away from x in the embedding space, how does the likelihood that the participant knows about the content at a given location fall off with distance? Conversely, suppose the participant instead answered that same question incorrectly. Again, as we move farther away from x in the embedding space, how does the likelihood that the participant does not know about the content at a given coordinate change with distance? We reasoned that, assuming our embedding space is capturing something about how individuals actually organize their knowledge, a participant’s ability to answer questions embedded very close to x should tend to be similar to their ability to answer the question embedded at x. But once we reach some sufficiently large distance from x, our ability to infer whether or not a participant will correctly answer a question based on their ability to answer the question at x should be no better than guessing based on their overall proportion of correctly answered questions. In other words, beyond the maximum distance at which a participant’s ability to answer the question at x is informative of their ability to answer a second question at location y, guessing the outcome at y based on the outcome at x should be no more successful than guessing based on a measure that does not consider embedding-space distance.

With these ideas in mind, we asked: conditioned on a participant’s ability to answer a given question correctly, what proportion of all questions within some radius r of its embedding coordinate were they able to answer correctly? We plotted this proportion as a function of r for questions that participants answered correctly, and for questions they answered incorrectly. As shown in Fig. 7, we found that quiz performance falls off smoothly with distance, and the rate at which it falls off does not appear to differ across quizzes, as measured by the distance at which performance becomes statistically indistinguishable from a simple proportion-correct score (see Estimating the “smoothness” of knowledge). This suggests that, at least within the region of text embedding space spanned by the questions our study’s participants answered (and as characterized using our topic model), the rate at which knowledge changes with distance is relatively constant, even as participants’ overall level of knowledge varies across quizzes and regions of the embedding space.

Knowledge estimates need not be limited to the contents of these particular lectures and quizzes. As illustrated in Fig. 8, our general approach to estimating knowledge from a small number of quiz questions may be extended to any content, given its text embedding coordinate. To visualize how knowledge “spreads” through text embedding space to content beyond the lectures participants watched and the questions they answered, we first fit a new topic model to the lectures’ sliding windows with k = 100 topics. Conceptually, increasing the number of topics used by the model functions to increase the “resolution” of the embedding space, providing a greater ability to estimate knowledge for content that is highly similar to (but not precisely the same as) that contained in the two lectures used to train the model. Aside from increasing the number of topics from 15 to 100, all other procedures and model parameters were carried over from the preceding analyses. As in our other analyses, we resampled each lecture’s topic trajectory to 1 Hz and projected each question into a shared text embedding space.

We projected the resulting 100-dimensional topic vectors (for each second of the lectures and each quiz question) onto a shared 2-dimensional plane (see Creating knowledge and learning map visualizations). Next, we sampled points from a 100 × 100 grid of coordinates that evenly tiled a rectangle enclosing the 2D projections of the lectures and questions. We then used Eq. (4) to estimate participants’ knowledge at each of these 10,000 sampled locations, and averaged these estimates across participants to obtain an estimated average “knowledge map” (Fig. 8A). Intuitively, the knowledge map constructed from a given quiz’s responses provides a visualization of how much participants knew about any content expressible by the fitted text embedding model at the point in time when they completed that quiz. We note that we used these 2D maps solely for visualization; all relevant comparisons, distance computations, and statistical tests we report above were carried out in the original 15-dimensional space, using the 15-topic model.

Several features of the resulting knowledge maps are worth noting. The average knowledge map estimated from Quiz 1 responses (Fig. 8A, leftmost map) shows that participants tended to have relatively little knowledge about any parts of the text embedding space (i.e., the shading is relatively dark everywhere). The knowledge map estimated from Quiz 2 responses shows a marked increase in knowledge on the left side of the map (around roughly the same range of coordinates traversed by the Four Fundamental Forces lecture, indicated by the dotted blue line). In other words, participants’ estimated increase in knowledge is localized to conceptual content that is nearby (i.e., related to) the content from the lecture they watched prior to taking Quiz 2. This localization is non-trivial: these knowledge estimates are informed only by the embedded coordinates of the quiz questions, not by the embeddings of either lecture (see Eq. (4)). Finally, the knowledge map estimated from Quiz 3 responses shows a second increase in knowledge, localized to the region surrounding the embedding of the Birth of Stars lecture participants watched immediately prior to taking Quiz 3.

Another way of visualizing these content-specific increases in knowledge after participants viewed each lecture is displayed in Fig. 8B. Taking the point-by-point difference between the knowledge maps estimated from responses to a successive pair of quizzes yields a “learning map” that describes the change in estimated knowledge from one quiz to the next. These learning maps highlight that the estimated knowledge increases we observed across maps were specific to the regions around the embeddings of each lecture, in turn.

Because the 2D projection we used to construct the knowledge and learning maps is invertible, we may gain additional insights into these maps’ meanings by reconstructing the original high-dimensional topic vector for any location on the map we are interested in. For example, this could serve as a useful tool for an instructor looking to better understand which content areas a student (or a group of students) knows well (or poorly). As a demonstration, we show the top-weighted words from the blends of topics reconstructed from three example locations on the maps (Fig. 8C): one point near the Four Fundamental Forces embedding (yellow), a second point near the Birth of Stars embedding (orange), and a third point between the two lectures’ embeddings (pink). As shown in the word clouds in Panel C, the top-weighted words at the example coordinate near the Four Fundamental Forces embedding tended to be weighted more heavily by the topics expressed in that lecture. Similarly, the top-weighted words at the example coordinate near the Birth of Stars embedding tended to be weighted more heavily by the topics expressed in that lecture. The top-weighted words at the example coordinate between the two lectures’ embeddings show a roughly even mix of words most strongly associated with each lecture.

Methods
Participants
We enrolled a total of 50 Dartmouth undergraduate students in our study. Participants received optional course credit for enrolling. No statistical method was used to predetermine sample size and no participants were excluded from the dataset. We asked each participant to complete a demographic survey that included questions about their age, gender, native spoken language, ethnicity, race, hearing, color vision, sleep, coffee consumption, level of alertness, and several aspects of their educational background and prior coursework.

Participants’ ages ranged from 18 to 22 years (mean: 19.52 years; standard deviation: 1.09 years). A total of 15 participants reported their gender as male and 35 participants reported their gender as female. As we had no a priori hypotheses regarding sex or gender differences in this study, neither sex nor gender was considered in the study design and we did not perform any sex- or gender-based analyses. A total of 49 participants reported their native language as “English” and 1 reported having another native language. A total of 47 participants reported their ethnicity as “Not Hispanic or Latino” and three reported their ethnicity as “Hispanic or Latino.” Participants reported their races as White (32 participants), Asian (14 participants), Black or African American (5 participants), American Indian or Alaska Native (1 participant), and Native Hawaiian or Other Pacific Islander (1 participant). (Note that some participants selected multiple racial categories.)

A total of 49 participants reporting having normal hearing and 1 participant reported having some hearing impairment. A total of 49 participants reported having normal color vision and 1 participant reported being colorblind. Participants reported having had, on the night prior to testing, 2–4 h of sleep (1 participant), 4–6 h of sleep (9 participants), 6–8 h of sleep (35 participants), or 8+ h of sleep (5 participants). They reported having consumed, on the same day and leading up to their testing session, 0 cups of coffee (38 participants), 1 cup of coffee (10 participants), 3 cups of coffee (1 participant), or 4+ cups of coffee (1 participant).

No participants reported that their focus was currently impaired (e.g., by drugs or alcohol). Participants reported their current level of alertness, and we converted their responses to numerical scores as follows: “very sluggish” (−2), “a little sluggish” (−1), “neutral” (0), “fairly alert” (1), and “very alert” (2). Across all participants, a range of alertness levels were reported (range: −2–1; mean: −0.10; standard deviation: 0.84).

Participants reported their undergraduate major(s) as “social sciences” (28 participants), “natural sciences” (16 participants), “professional” (e.g., pre-med or pre-law; 8 participants), “mathematics and engineering” (7 participants), “humanities” (4 participants), or “undecided” (3 participants). Note that some participants selected multiple categories for their undergraduate major(s). We also asked participants about the courses they had taken. In total, 45 participants reported having taken at least one Khan Academy course in the past, and 5 reported not having taken any Khan Academy courses. Of those who reported having watched at least one Khan Academy course, 7 participants reported having watched 1–2 courses, 11 reported having watched 3–5 courses, 8 reported having watched 5–10 courses, and 19 reported having watched 10 or more courses. We also asked participants about the specific courses they had watched, categorized under different subject areas. In the “Mathematics” area, participants reported having watched videos on AP Calculus AB (21 participants), Precalculus (17 participants), Algebra 2 (14 participants), AP Calculus BC (12 participants), Trigonometry (11 participants), Algebra 1 (10 participants), Geometry (8 participants), Pre-algebra (7 participants), Multivariable Calculus (5 participants), Differential Equations (5 participants), Statistics and Probability (4 participants), AP Statistics (2 participants), Linear Algebra (2 participants), Early Math (1 participant), Arithmetic (1 participant), and other videos not listed in our survey (5 participants). In the “Science and engineering” area, participants reported having watched videos on Chemistry, AP Chemistry, or Organic Chemistry (21 participants); Physics, AP Physics I, or AP Physics II (18 participants); Biology, AP Biology, or High school Biology (15 participants); Health and Medicine (1 participant); and other videos not listed in our survey (5 participants). We also asked participants whether they had specifically seen the videos used in our experiment. Of the 45 participants who reported having having taken at least one Khan Academy course in the past, 44 participants reported that they had not watched the Four Fundamental Forces video and 1 participant reported that they were not sure whether they had watched it. All participants reported that they had not watched the Birth of Stars video. When we asked participants about non-Khan Academy online courses, they reported having watched or taken courses on Mathematics (15 participants), Science and engineering (11 participants), Test preparation (9 participants), Economics and finance (3 participants), Arts and humanities (2 participants), Computing (2 participants), and other categories not listed in our survey (17 participants). Finally, we asked participants about in-person courses they had taken in different subject areas. They reported taking courses in Mathematics (38 participants), Science and engineering (37 participants), Arts and humanities (34 participants), Test preparation (27 participants), Economics and finance (26 participants), Computing (14 participants), College and careers (7 participants), and other courses not listed in our survey (6 participants).

Experiment
We hand-selected two course videos from the Khan Academy platform: Four Fundamental Forces (an introduction to gravity, electromagnetism, the weak nuclear force, and the strong nuclear force; duration: 10 min and 29 s) and Birth of Stars (an introduction to how stars are formed; duration: 7 min and 57 s). All participants viewed the videos in the same order (i.e., Four Fundamental Forces followed by Birth of Stars).

We then hand-created 39 multiple-choice questions: 15 about the conceptual content of Four Fundamental Forces (i.e., Lecture 1), 15 about the conceptual content of Birth of Stars (i.e., Lecture 2), and 9 questions that tested for general conceptual knowledge about basic physics (covering material that was not presented in either video). To help broaden the set of lecture-specific questions, our team worked through each lecture in small segments to identify what each segment was about conceptually, and then write a question about that concept. The general physics questions were drawn from our team’s prior coursework and areas of interest, along with internet searches and brainstorming with the project team and other members of J.R.M.’s lab. Although we attempted to design the questions to test “conceptual knowledge,” we note that estimating the specific “amount” of conceptual understanding that each question requires to answer is somewhat subjective, and might even come down to the strategy a given participant used to answer the question at that particular moment. The full set of questions and answer choices may be found in Supplementary Table 1. The final set of questions (and response options) was reviewed and approved by J.R.M. before we collected or analyzed the text or experimental data.

Over the course of the experiment, participants completed three 13-question multiple-choice quizzes: the first before viewing Lecture 1, the second between Lectures 1 and 2, and the third after viewing Lecture 2 (see Fig. 1). The questions appearing on each quiz, for each participant, were randomly chosen from the full set of 39, with the constraints that (a) each quiz contained exactly 5 questions about Lecture 1, 5 questions about Lecture 2, and 3 questions about general physics knowledge, and (b) each question appeared exactly once for each participant. The orders of questions on each quiz, and the orders of answer options for each question, were also randomized. We obtained informed consent from all participants, and our experimental protocol was approved by the Committee for the Protection of Human Subjects at Dartmouth College. We used this experiment to develop and test our computational framework for estimating knowledge and learning.

Analysis
Statistics. All of the statistical tests performed in our study were two-sided. The 95% confidence intervals we report for each correlation were estimated from bootstrap distributions of 10,000 correlation coefficients obtained by sampling (with replacement) from the observed data.

Constructing text embeddings of multiple lectures and questions. We adapted an approach we developed in prior work34 to embed each moment of the two lectures and each question in our pool in a common representational space. Briefly, our approach uses a topic model (Latent Dirichlet Allocation18) trained on a set of documents to discover a set of k “topics” or “themes.” Formally, each topic is defined as a distribution of weights over words in the model’s vocabulary (i.e., the union of all unique words across all documents, excluding stopwords). Conceptually, each topic is intended to give larger weights to words that are semantically related (as inferred from their tendency to co-occur in the same document). After fitting a topic model, each document in the training set, or any new document that contains at least some of the words in the model’s vocabulary, may be represented as a k-dimensional vector describing how much that document (most probably) reflects each topic. To select an appropriate k for our model, as a starting point, we identified the minimum number of topics that yielded at least one “unused” topic (i.e., in which all words in the vocabulary were assigned uniform weights) after training. This indicated that the number of topics was sufficient to capture the set of latent themes present in the two lectures (from which we constructed our document corpus, as described below). We found this value to be k = 15 topics. We found that with a limited number of additional adjustments following ref. 61, such as removing corpus-specific stopwords, the model yielded (subjectively) sensible and coherent topics. The distribution of weights over words in the vocabulary for each discovered topic is shown in Supplementary Fig. 1, and each topic’s top-weighted words may be found in Supplementary Table 2.

As illustrated in Fig. 2A, we started by building up a corpus of documents using overlapping sliding windows that spanned each lecture’s transcript. Khan Academy provides professionally created, manual transcriptions of all lecture videos for closed captioning. However, such transcripts would not be readily available in all contexts to which our framework could potentially be applied. Khan Academy videos are hosted on the YouTube platform, which additionally provides automated captions. We opted to use these automated transcripts (which, in prior work, we have found to be of sufficiently near-human quality to yield reliable data in behavioral studies62) when developing our framework in order to make it more directly extensible and adaptable by others in the future.

We fetched these automated transcripts using the youtube-transcript-api Python package63. Each transcript consisted of one timestamped line of text for every few seconds (mean: 2.34s; standard deviation: 0.83s) of spoken content in the lecture (i.e., corresponding to each individual caption that would appear on-screen if viewing the lecture via YouTube, and when those lines would appear). We defined a sliding window length of (up to) w = 30 transcript lines and assigned each window a timestamp corresponding to the midpoint between the timestamps for its first and last lines. This w parameter was chosen to match the same number of words per sliding window (rounded to the nearest whole word, and before preprocessing) as the sliding windows we defined in our prior work (ref. 34; i.e., 185 words per sliding window).

These sliding windows ramped up and down in length at the beginning and end of each transcript, respectively. In other words, each transcript’s first sliding window covered only its first line, the second sliding window covered the first two lines, and so on. This ensured that each line from the transcripts appeared in the same number (w) of sliding windows. We next performed a series of standard text preprocessing steps: normalizing case, lemmatizing, removing punctuation and removing stop-words. We constructed our corpus of stop-words by augmenting the Natural Language Toolkit (NLTK64) English stop-word list with the following additional words, selected using one of the approaches suggested by ref. 61: “actual,” “actually,” “also,” “bit,” “could,” “e,” “even,” “first,” “follow,” “following,” “four,” “let,” “like,” “mc,” “really,” “saw,” “see,” “seen,” “thing,” and “two.” This yielded sliding windows containing an average of 73.8 remaining words, and spanning an average of 62.22 seconds. We treated the text from each sliding window as a single document and combined these documents across the two lectures’ windows to create a single training corpus for the topic model.

After fitting the topic model to the two lectures’ transcripts, we could use the trained model to transform arbitrary (potentially new) documents into k-dimensional topic vectors. A convenient property of these topic vectors is that documents that reflect similar blends of topics (i.e., documents that reflect similar themes, according to the model) will yield similar coordinates (in terms of correlation, cosine similarity, Kullback-Leibler divergence, Euclidean distance, or other geometric measures). In general, the similarity between different documents’ topic vectors may be used to characterize the similarity in conceptual content between the documents.

We transformed each sliding window’s text into a topic vector, and then used linear interpolation (independently for each topic dimension) to resample the resulting time series to one vector per second. We also used the fitted model to obtain topic vectors for each quiz question in our pool (see Supplementary Table 1). Taken together, we obtained a trajectory for each lecture video describing its path through topic space, and a single coordinate for each question (Fig. 2C). Embedding both lectures and all of the questions using a common model enables us to compare the content from different moments of the lectures, compare the content across lectures, and estimate potential associations between specific questions and specific moments of lecture content.

Estimating dynamic knowledge traces. We used the following equation to estimate each participant’s knowledge about timepoint t of a given lecture, kb(t):

kb(f(t,L)) = [sum_{i in correct} ncorr(f(t,L), f(i,Q))] / [sum_{j=1}^N ncorr(f(t,L), f(j,Q))] (1)

where

ncorr(x,y) = (corr(x,y) - mincorr) / (maxcorr - mincorr) (2)

and where mincorr and maxcorr are the minimum and maximum correlations between the topic vectors for any lecture timepoint and quiz question, taken over all timepoints in the given lecture and all questions about that lecture appearing on the given quiz. We also define f(s,Ω) as the sth topic vector from the set of topic vectors Ω. Here t indexes the time series of lecture topic vectors L, and i and j index the topic vectors of questions Q used to estimate the participant’s knowledge. Note that “correct” denotes the set of indices of the questions the participant answered correctly on the given quiz.

Intuitively, ncorr(x, y) is the correlation between two topic vectors (e.g., the topic vector x for one timepoint in a lecture and the topic vector y for one question on a quiz), normalized by the minimum and maximum correlations (across all timepoints and questions j) to range between 0 and 1, inclusive. Equation (1) then computes the weighted average proportion of correctly answered questions about the content presented at timepoint t, where the weights are given by the normalized correlations between timepoint t’s topic vector and the topic vectors for each question. The normalization step (i.e., using ncorr instead of the raw correlations) ensures that every question contributes some non-negative amount to the knowledge estimate.

Generalized linear mixed models. In the set of analyses reported in Fig. 6, we assessed whether estimates of participants’ knowledge at the embedding coordinates of individual quiz questions could be used to reliably predict their abilities to correctly answer those questions. In essence, we treated each question a given participant answered on a given quiz as a lecture consisting of a single time point, and used Eq. (1) to estimate the participant’s knowledge for its embedding coordinate based on their performance on all other questions they answered on that same quiz (“All questions”; Fig. 6, top row). Additionally, for each lecture-related question (i.e., excluding questions about general physics knowledge), we computed analogous knowledge estimates based on two different subsets of questions the participant answered on the same quiz: (1) all other questions about the same lecture as the target question (“Within-lecture”; Fig. 6, middle rows), and (2) all questions about the other of the two lectures (“Across-lecture”; Fig. 6, bottom rows).

In performing these analyses, our null hypothesis is that the knowledge estimates we compute based on the quiz questions’ embedding coordinates do not provide useful information about participants’ abilities to correctly answer those questions—in other words, that there is no meaningful difference (on average) between the knowledge estimates we compute for questions participants answered correctly versus incorrectly. Specifically, since we estimate knowledge for a given embedding coordinate as a weighted proportion-correct score (where each question’s weight reflects its embedding-space distance from the target coordinate; see Eq. (1)), if these weights are uninformative (e.g., randomly distributed), then our estimates of participants’ knowledge should be equivalent (on average) to the unweighted proportion of correctly answered questions used to compute them. In general, for a given participant and quiz, this expected null value (i.e., that participant’s proportion-correct score on that quiz) is the same for any coordinate in the embedding space (e.g., any lecture timepoint, quiz question, etc.). However, in the “All questions” and “Within-lecture” versions of the analyses shown in Fig. 6, we estimate each participant’s knowledge for each target question using all other questions (or all other questions about the same lecture) they answered on that participant’s quiz. This introduces a dependency between the participant’s proportion-correct score on the remaining questions available to estimate their knowledge for it. For example, suppose a participant correctly answered n out of q questions on a given quiz. If we hold out a single correctly answered question as the target, the proportion of remaining questions answered correctly would be (n-1)/(q-1), whereas if we hold out a single incorrectly answered question, the proportion of remaining questions answered correctly would be n/(q-1). Thus, the proportion of correctly answered remaining questions (and therefore the null-hypothesized value of a knowledge estimate computed from them) is always lower for target questions a participant answered correctly than for those they answered incorrectly.

To correct for this baseline difference under our null hypothesis, we used a rebalancing procedure that ensured our knowledge estimates for questions each participant answered correctly and incorrectly were computed from the same proportion of correctly answered questions. For each target question on a given participant’s quiz, we first identified all remaining questions with the opposite “correctness” label (i.e., if the target question was answered correctly, we identified all remaining incorrectly answered questions, and vice versa). We then held out each of these opposite-label questions, in turn, along with the target question, and estimated the participant’s knowledge for the target question using all other remaining questions. Since each of these subsets of remaining questions was constructed by holding out one correctly answered question and one incorrectly answered question from the participant’s quiz responses, if the participant correctly answered n out of q questions total, then their proportion-correct score on each subset of questions used to estimate their knowledge would be (n-1)/(q-2), regardless of whether they answered the target question correctly or incorrectly. Finally, we averaged over these per-subset knowledge estimates to obtain a rebalanced estimate of the participant’s knowledge for the target question that leveraged information from all remaining questions’ embedding coordinates, but whose expected value under our null hypothesis was the same as that of each individual subset (n-1)/(q-2). By equalizing the null-hypothesized values of knowledge estimates for correctly and incorrectly answered questions, this procedure ensures that any meaningful relationships we observe between participants’ estimated knowledge for individual quiz questions and their abilities to correctly answer them reflect the predictive power of the embedding-space distances we use to weight questions’ contributions to the knowledge estimates, rather than an artifact of our testing procedure. Note that if a participant answered all or no questions on a given quiz correctly, their responses contained no opposite-label questions with which to perform this rebalancing, and we therefore excluded their data from our analyses for that quiz. We used this rebalancing procedure when constructing knowledge estimates for the “All questions” and “Within-lecture” versions of the analyses shown in Fig. 6, but not for the “Across-lecture” analyses as, in this case, the target questions and the questions used to estimate participants’ knowledge for them were drawn from different subsets of quiz questions (those about one lecture, and those about the other), and were therefore independent.

In each version of this analysis (i.e., row in Fig. 6), and separately for each of the three quizzes (i.e., column in Fig. 6), we then fit a generalized linear mixed model (GLMM) with a logistic link function to the set of knowledge estimates for all questions (or all questions about a particular lecture) that participants answered on the given quiz. We implemented these models in R using the lme4 package65 and fit them according to the guidance in refs. 66,67. Specifically, we initially fit each model with the maximal random effects structure afforded by our design, which we identified as:

accuracy ~ knowledge + (knowledge | participant) + (knowledge | question)

where accuracy is a binary value indicating whether each target question was answered correctly or incorrectly, knowledge is estimated knowledge at each target question’s embedding coordinate, participant is a unique identifier assigned to each participant, and question is a unique identifier assigned to each quiz question. For models we fit using knowledge estimates for target questions about multiple content areas (i.e., in the “All questions” version of the analysis), we also included an additional random effect term, (knowledge | lecture), where lecture is a categorical value denoting whether the target question was about Four Fundamental Forces, Birth of Stars, or general physics knowledge. Note that with our coding scheme, identifiers for each question are implicitly nested within levels of lecture and so do not require explicit nesting in our model formula. We then iteratively removed random effects from the maximal model until it successfully converged with a full-rank random effects variance-covariance matrix. We obtained the odds ratios reported in Fig. 6 by exponentiating the estimated coefficient for knowledge from each fitted model. Conceptually, these odds ratios represent how many times greater the odds are that a given participant will answer a given question correctly if their estimated knowledge for its embedding coordinate is 1, compared to if it is 0. We estimated 95% confidence intervals for each odds ratio by generating 10,000 random subsamples (of full size, with replacement) from the data used to fit each model, and refitting the models to each subsample to obtain bootstrap distributions of 10,000 odds ratios.

To assess the predictive value of our knowledge estimates, we compared each GLMM’s ability to explain participants’ success on individual quiz questions to that of an analogous model which assumed (as we assume under our null hypothesis) that knowledge estimates for correctly and incorrectly answered questions did not systematically differ, on average. Specifically, we used the same sets of observations to which we fit each “full” model to fit a second “null” model with the same random effects structure, but with the coefficient for the fixed effect of knowledge constrained to zero (i.e., we removed this term from the null model). We then compared each full model to its reduced (null) equivalent using a likelihood-ratio test (LRT). Because the standard asymptotic χ2 approximation of the null distribution for the LRT statistic (λLR) can be anti-conservative for finite sample sizes68–70, we computed p-values for these tests using a parametric bootstrap procedure71,72. For each of 10,000 bootstraps, we used the fitted null model to simulate a sample of observations of equal size to our original sample. We then re-fit both the null and full models to this simulated sample and compared them via an LRT. This yielded a distribution of λLR statistics we may expect to observe given data that conforms to our null hypothesis. We computed a corrected p-value for our observed λLR as (r+1)/(n+1), where r is the number of simulated model comparisons that yielded a λLR greater than our observed value and n is the number of simulations we ran (10,000).

Estimating the “smoothness” of knowledge. In the analysis reported in Fig. 7A, we show how participants’ ability to correctly answer quiz questions changes as a function of distance from a given correctly or incorrectly answered reference question. We used a bootstrap-based approach to estimate the maximum distances over which these proportions of correctly answered questions could be reliably distinguished from participants’ overall average proportion of correctly answered questions.

For each of 10,000 iterations, we drew a random subsample (with replacement) of 50 participants from our dataset. Within each iteration, we first computed the 95% confidence interval of the across-subsample-participants mean proportion correct on each of the three quizzes, separately. To compute this interval for each quiz, we repeatedly (1000 times) subsampled participants (with replacement, from the outer subsample for the current iteration) and computed the mean proportion correct of each of these inner subsamples. We then identified the 2.5th and 97.5th percentiles of the resulting distributions of 1,000 means. These three intervals (one for each quiz) served as our thresholds for confidence that the proportion correct within a given distance from a reference question was reliably different (at the p < 0.05 significance level) from the average proportion correct across all questions on the given quiz.

Next, for each participant in the current subsample, and for each of the three quizzes they completed (separately), we iteratively treated each of the 15 questions appearing on the given quiz as the reference question. We constructed a series of concentric 15-dimensional spheres centered on the reference question’s embedding-space coordinate, where each successive sphere’s radius increased by 0.01 (correlation distance) between 0 and 2, inclusive (i.e., tiling the range of possible correlation distances with 201 spheres in total). We then computed the proportion of questions enclosed within each sphere that the participant answered correctly, and averaged these per-radius proportion-correct scores across reference questions that were answered correctly, and those that were answered incorrectly. This resulted in two number-of-spheres sequences of proportion-correct scores for each subsample participant and quiz: one derived from correctly answered reference questions, and one derived from incorrectly answered reference questions.

We computed the across-subsample-participants mean proportion correct for each radius value (i.e., sphere) and “correctness” of reference question. This yielded two sequences of proportion-correct scores for each quiz, analogous to the blue and red lines displayed in Fig. 7A, but for the present subsample. For each quiz, we then found the minimum distance from the reference question (i.e., sphere radius) at which each of these two sequences of per-radius proportion-correct scores intersected the 95% confidence interval for the overall proportion correct (i.e., analogous to the black error bands in Fig. 7A). This resulted in two intersection distances for each quiz (for correctly answered and incorrectly answered reference questions). Repeating this full process for each of the 10,000 bootstrap iterations output two distributions of intersection distances for each of the three quizzes. The means and 95% confidence intervals for these distributions are plotted in Fig. 7B.

Creating knowledge and learning map visualizations. An important feature of our approach is that, given a trained text embedding model and participants’ performance on each quiz question, we can estimate their knowledge about any content expressible by the embedding model—not solely the content explicitly probed by the quiz questions, or even appearing in the lectures. To visualize these estimates (Fig. 8, Supplementary Figs. 5, 6, 7, 8, and 9), we used Uniform Manifold Approximation and Projection (UMAP73,74) to construct a 2D projection of the text embedding space. Whereas our main analyses used a 15-topic embedding space, we used a 100-topic embedding space for these visualizations. This change in the number of topics overcame an undesirable behavior in the UMAP embedding procedure, whereby embedding coordinates for the 15-topic model tended to be grouped into separated clusters, rather than forming a smooth trajectory through the 2D space. When we increased the number of topics to 100, the embedding coordinates in the 2D space formed a smooth trajectory through the space, with substantially less aggressive grouping (Fig. 8). Creating a “map” by sampling this 100-dimensional space at high resolution to obtain an adequate set of topic vectors spanning the embedding space would be computationally intractable. However, sampling a 2D grid is trivial.

At a high level, the UMAP algorithm obtains low-dimensional embeddings by minimizing the cross-entropy between the pairwise (clustered) distances between the observations in their original (e.g., 100-dimensional) space and the pairwise (clustered) distances in the low-dimensional embedding space (in our approach, the embedding space is 2D). In our implementation, pairwise distances in the original high-dimensional space were defined as 1 minus the correlation between each pair of coordinates, and pairwise distances in the low-dimensional embedding space were defined as the Euclidean distance between each pair of coordinates.

In our application, all of the coordinates we embedded were topic vectors, whose elements are always non-negative and sum to one. Although UMAP is an invertible transformation at the embedding locations of the original data, other locations in the embedding space will not necessarily follow the same implicit rules as the original high-dimensional data. For example, inverting an arbitrary coordinate in the embedding space might result in negative-valued vectors, which are incompatible with the topic modeling framework. To protect against this issue, we log-transformed the topic vectors prior to embedding them in the 2D space. When we inverted the embedded vectors (e.g., to estimate topic vectors for word clouds, as in Fig. 8C), we passed the inverted (log-transformed) values through the exponential function to obtain a vector of non-negative values, and normalized them to sum to one.

After embedding both lectures’ topic trajectories and the topic vectors of every question, we defined a rectangle enclosing the 2D projections of the lectures’ and quizzes’ embeddings. We then sampled points from a regular 100 × 100 grid of coordinates that evenly tiled this enclosing rectangle. We sought to estimate participants’ knowledge (and learning; i.e., changes in knowledge) at each of the resulting 10,000 coordinates.

To generate our estimates, we placed a set of 39 radial basis functions (RBFs) throughout the embedding space, centered on the 2D projections for each question (i.e., we included one RBF for each question). At coordinate x, the value of an RBF centered on a question’s coordinate μ is given by:

RBF(x,μ,λ) = exp(- ||x - μ||^2 / λ) (3)

The λ term in the RBF equation controls the smoothness of the function, where larger values of λ result in smoother maps. In our implementation, we used λ = 50. Next, we estimated the “knowledge” at each coordinate x, using:

kb(x) = [sum_{i in correct} RBF(x,qi,λ)] / [sum_{j=1}^N RBF(x,qj,λ)] (4)

Equation (4) computes the weighted proportion of correctly answered questions, where the weights are given by how nearby (in the 2D space) each question is to x. We also defined “learning maps” as the coordinate-by-coordinate differences between any pair of knowledge maps. Intuitively, learning maps reflect the change in knowledge between two maps.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "Fitzpatrick and colleagues present a neat method for quantifying the content of real-world educational lecture videos and show that the resulting embedding space can be used to better understand and visualize quiz results assessing what knowledge students have learned. This manuscript is well-written and the figures are well-designed; I’m enthusiastic about the approach and the results are compelling. I have a couple clarifying questions about the methods, but my main concerns revolve around making sure the interpretations match the scope of the present data.\n1. I have two related high-level comments, both of which I’m having hard time articulating. First, I think the authors should be careful about the interpretative language they use throughout. Sometimes the language feels a bit lofty or overreaching. Let me try to provide an example: I’m not sure I fully buy the rhetorical parallel between “memorization” and “understanding.” I don’t think anyone in educational contexts really thinks of rote memorization as memorizing character strings the way a language model might. On the other hand, I don’t really see how the topic model used in the present study (or even much more sophisticated language models) are doing anything above and beyond memorizing statistical (i.e. co-occurrence) structure (albeit at multiple scales). Sure, the model is not a simple, discrete lookup table, but I’m not sure we want to make any strong claim that the model captures “understanding” of physics either. If I understand correctly, the main advance from “memorization” toward “understanding” is made by embedding words in a continuous vector space, granting Shepard-(1987) style(?) generalization (i.e. the authors commitment that knowledge is fundamentally “smooth”). But the analyses simply demonstrate that the embeddings for a test set of words (i.e. quiz questions) can be interpolated from the embedding space learned from a training set of words (i.e. the video transcripts). Is this kind of interpolation fundamentally different from “memorization”? (A more radical claim here might be that much of what is seemingly “understanding” can be reduced to this kind of interpolation—which I generally agree with, cf. our Hasson et al., 2020 paper—but I doubt the authors really want to make that claim either.)\n2a. Second, I think the authors should be very clear about the limits/scope of their approach, particularly based on the dataset used herein. For instance, in the Introduction, you use aspirational language about “the sum total of a person’s knowledge” and “the nearly infinite set of possible things a person could know”—but the actual data/model used here is relatively small-scale (and there’s no explicit evidence for scalability). I can’t imagine that the co-occurrence statistics across only <20 minutes of video lectures are particularly dense or rich. Just to confirm, the model is not pretrained in any way and does not incorporate any conceptual information from outside the lecture videos, right? For example, the authors say (line 492) “Conceptually, each topic is intended to give larger weights to words that are semantically related *or* tend to co-occur in the same documents” (my emphasis on “or”). Does the model know that words are semantically related above and beyond the co-occurrence statistics? My understanding was that “semantically related” is derived entirely from the co-occurrences statistics. In another example, you say “Knowledge estimates need not be limited to the content of the lectures.” But the embedding space is strictly limited to the co-occurrence structure extracted from these particular lectures—right?\n2b. Put another way, there’s a tension in the writing between the potential generality of the method and the targeted nature of the embedding space (i.e. tuned to the content of the two lectures) used herein. Here’s a specific example: In my understanding, the negative correlations between mismatching lecture and video topic weights (line 176) are an indicator that the embedding space reflects an “either this or that” structure specific to the two lectures. I have the intuition that near-zero correlation here would provide stronger evidence for “specificity” (line 179) than anticorrelation (as we see with the general physics questions). I wonder if this trivializes the dimensionality reduction results in Figure 7—i.e. could it be any other way? (Maybe this is fine!)\n2c. Another specific example: The authors want to make a strong claim that the model doesn’t rely on surface-level similarity and that the “specific words” used in the quiz questions have “very little overlap” with the lectures? But there must be some overlap (i.e. co-occurrence), right? In my understanding, a completely disjoint set of words, none of which occurred in these video transcripts, would not map onto this embedding space in any meaningful way. To be clear, I don’t think this is really a problem or a weakness of the method!—I’m just trying to articulate some discomfort when reading.\n3. How exactly were the questions constructed for each lecture? For example, Figure 2 suggests that the questions follow the trajectories of the lectures, at least to some extent; obviously this is sensible and “by design.” It’s great that the authors provide the full set of questions/answers (Table S1)! However, I think it’s important that the authors describe their thought process in designing the questions a bit more explicitly. For example, how exactly did they minimize “surface-level” similarity while ensuring enough conceptual overlap that the questions map onto the embedding space in a meaningful way?\n4. The authors correlated the topic weights for each question with the topic weights at each time point in the lecture, and in Figure 4 show that “each question appeared to be temporally specific” (line 208). Intuitively, this seems reasonable… But couldn’t this result also be interpreted as creating tension with the idea that the questions aren’t simply keying into “surface-level” similarities? Put another way, if I thought this approach captured some kind of holistic “understanding”, I might hypothesize that good questions would integrate multiple concepts introduced across different parts of the lecture. (I might be conflating the overall validity of the approach for capturing “understanding” with the quality of the questions here.)\n5. The authors used windows comprising 30 lines(?) of the transcript to define documents for input to the topic model. This seems like an important parameter as it impacts the co-occurrence structure observed by the topic model, right? How was this number 30 chosen? I assume it represents some happy medium between providing a wide enough window to yield rich co-occurrence structure while retaining some temporal specificity. Transcript “lines” seems like a somewhat arbitrary unit; I assume it relates to how YouTube renders the transcript? How many words tend to occur in each of these windows? What’s the average duration of these windows in seconds?\n6. I don’t fully understand the motivation for choosing 15 topics for most of the analyses, or why you increase to 100 topics for Figure 7. I don’t think there’s anything wrong with picking a somewhat arbitrary round number in these cases; just curious if there’s any particular reasoning behind the present choices.\nMinor comments:\nRegarding my previous comments on scalability, do the authors think this kind of approach could be extended from topic models to the contextual embeddings learned by large language models? Could be worth mentioning if this is a valid future direction, given the current popularity of these models. My guess is that if we want a method that can capture “the sum total of a person’s knowledge” and “the nearly infinite set of possible things a person could know,” large language models are currently the best approach.\nIn Figures 2 and 7, you use PCA to visualize two- and three-dimensional projections of the embedding space. What proportion of cumulative variance do these first two and three PCs account for?\nAll participants saw the Four Fundamental Forces video first, followed by the Birth of a Star video, right? Should we be worried about some order effect?\nLine 283: These t-values have 600+ degrees of freedom; I assume this is determined by the total number of questions across all participants. Should we be worried that this kind of statistical test doesn’t take into account subject-level variability?\nLine 492: Do you use a specific set of stop words? (e.g. the NLTK package has a standard list of stop words)\nLine 557: “insure” > “ensure”\nReferences:\nHasson, U., Nastase, S. A., & Goldstein, A. (2020). Direct fit to nature: an evolutionary perspective on biological and artificial neural networks. Neuron, 105(3), 416-434. https://doi.org/10.1016/j.neuron.2019.12.002\nShepard, R. N. (1987). Toward a universal law of generalization for psychological science. Science, 237(4820), 1317–1323. https://doi.org/10.1126/science.3629243", "reviewer_2": "The authors develop a framework for characterizing the acquisition of knowledge across time, specifically in the context of educational lectures. Participants watched two physics-related lectures on the Khan Academy platform, and completed quizzes before the first lecture, in between lectures, and after the second lecture. Questions on all quizzes tapped into knowledge from the two distinct lectures (as well as general physics knowledge). The information contained in the lectures and associated quiz questions was embedded in a topic space derived from word co-occurrences contained therein. This approach enabled the authors to (1) define each of the two lectures as a trajectory through this embedding space, (2) quantitatively determine the differences in conceptual content between the two lectures (by assessing topic weight and variability), (3) match quiz questions to specific moments in the lecture with relatively high resolution, (4) determine how much knowledge participants had about different points in the lecture, (5) predict whether a participant got a question correct or incorrect based on their estimated knowledge at that relevant coordinate in the embedding space, and (5) visualize how knowledge spreads through the embedding space as a result of watching the lectures. I am convinced that this framework yields more informative feedback on students’ quiz results, relative to calculating percent correct. I find the authors’ approach to be an exciting one, and that it provides methods others can use to quantify and visualize the specific content of one’s knowledge and how it might change over time. The methodology is sound, and the introduction and discussion are written well. This work primarily proposes a methodological framework rather than presenting new empirical results. Despite my overall enthusiasm, some other comments/suggestions are summarized below.\n1. It is mentioned multiple times that correspondences between quiz questions and lectures (e.g., localizing each quiz question to a specific point along a lecture’s trajectory) are not superficial, because “the lectures and questions used different words.” As far as I can see, there is no quantification of word overlap between lectures and quizzes, and we are supposed to take the authors’ word on this. I find it difficult to believe that there is no word overlap between lectures and quiz questions, and would like to see some quantified measure of word overlap. The degree to which correspondences in the embedding space are “superficial” or “deep” is unclear and as it stands, I am not fully convinced that the current analyses tap into deep conceptual knowledge.\n2. I would like to see the “Estimating Knowledge” analyses to go a step farther. The authors estimate participants’ knowledge at the position of each quiz question’s coordinate in a leave-one-out fashion, and show that these estimates reliably track whether the question was answered correctly or incorrectly. However, this analysis currently includes within-lecture prediction e.g., Lecture 1 quiz question performance to estimate a left-out Lecture 1 question. This analysis would be much more convincing and empirically interesting if a cross-lecture approach was used. That is, if for Quiz 2, performance on Lecture 1 questions could be used to predict accuracy on (unwatched) Lecture 2 quiz questions. This would be more convincing evidence that the embedding space is capturing deep conceptual meaning.\n3. In the current “Estimating Knowledge” analyses, I’m having a hard time understanding what is happening in Quiz 3. It makes sense that there’s no difference in the correct/incorrect distributions in Quiz 1, and it’s great that it emerges in Quiz 2, but I would have also expected to see the effect for Quiz 3 (and indeed a weak effect was found). The authors explain this as a “ceiling effect”, but it doesn’t look like the knowledge estimates are at ceiling in Fig6-Quiz3. Further, if the knowledge estimates based on the embedding space are not sufficiently predicting quiz question accuracy, then what does? In other words, if the knowledge estimates for all regions of the space were high, then why are the participants’ responses wrong? Is the model not capturing other pertinent knowledge, or do the authors think participants were choosing answers randomly?\n4. I don’t find the “Moment-by-moment” plots in Figure 5 to be useful, and in a sense they are misleading. With time on the x-axis, the implication is that the plot will be showing learning over time, as the lecture unfolds. This leads to the initial question of why the plots don’t show an increase over time. However, this analysis does not reveal increasing knowledge over time, but rather the degree of knowledge relating to each moment in the lecture, assessed after the lecture was completed. What extra information are we getting from these moment-to-moment plots, that we’re not getting from the averages presented in Fig 5B and 5D? Especially since there’s no indication of how topics are changing from moment to moment, these plots seem superfluous. An alternate possibility is to show what topics participants know in each of the quizzes.\n5. The Topic Weight Variability analysis (and corresponding Fig 3A) feels a bit lackluster. The authors have set up this rich, high-dimensional space in which to perform their other analyses, so it would be more satisfying if this analysis was also done using a geometric approach. For example, could variance be captured by the distance spanned by a lecture’s trajectory in each of the 15 dimensions? Could variability be visualized in 3D space? I also think the decision to visualize the topic variance, rather than topic weights, could be better justified—it’s not clear in the manuscript’s current form why we should care more about variance than weights, which is more intuitively relevant.\n6. This a relatively minor qualm, but referring to a coordinate in the embedding space as “a single concept” or “individual concept” doesn’t seem quite right (e.g., “We wondered whether our modeling framework might enable us to…infer participants’ knowledge at the scale of individual concepts (e.g., as captured by a single moment of a lecture). The notion of a single concept implies that concepts are units of some kind, and capturing conceptual knowledge in a high-dimensional space runs counter to that view. Further, what does a “single concept” mean if the authors count “fish”, “water” and “fish swim in water” to each be a concept (p. 3)? In this case, a full sentence would count as a concept, or even the information contained in a whole document. Maybe the authors take this view, but it feels particularly odd given the initial use of “fish” and “water” as initial examples.", "reviewer_3": "See attached file."}, "author_response": "Fitzpatrick and colleagues present a neat method for quantifying the content of real-world educational lecture videos\nandshowthattheresultingembeddingspacecanbeusedtobe erunderstandandvisualizequizresultsassessing\nwhatknowledgestudentshavelearned.Thismanuscriptiswell-wri enandthefiguresarewell-designed;I’m\nenthusiasticabouttheapproachandtheresultsarecompelling.Ihaveacoupleclarifyingquestionsaboutthe\nmethods,butmymainconcernsrevolvearoundmakingsuretheinterpretationsmatchthescopeofthepresentdata.\nWethankthereviewerfortheirpositiveassessment!\n1.Ihavetworelatedhigh-levelcomments,bothofwhichI’mhavinghardtimearticulating.First,Ithinktheauthors\nshouldbecarefulabouttheinterpretativelanguagetheyusethroughout.Sometimesthelanguagefeelsabitloftyor\noverreaching.Letmetrytoprovideanexample:I’mnotsureIfullybuytherhetoricalparallelbetween\n“memorization”and“understanding.”Idon’tthinkanyoneineducationalcontextsreallythinksofrote\nmemorizationasmemorizingcharacterstringsthewayalanguagemodelmight.Ontheotherhand,Idon’treally\nseehowthetopicmodelusedinthepresentstudy(orevenmuchmoresophisticatedlanguagemodels)aredoing\nanythingaboveandbeyondmemorizingstatistical(i.e.co-occurrence)structure(albeitatmultiplescales).Sure,the\nmodelisnotasimple,discretelookuptable,butI’mnotsurewewanttomakeanystrongclaimthatthemodel\ncaptures“understanding”ofphysicseither.IfIunderstandcorrectly,themainadvancefrom“memorization”\ntoward“understanding”ismadebyembeddingwordsinacontinuousvectorspace,grantingShepard-(1987)\nstyle(?)generalization(i.e.theauthorscommitmentthatknowledgeisfundamentally“smooth”).Buttheanalyses\nsimplydemonstratethattheembeddingsforatestsetofwords(i.e.quizquestions)canbeinterpolatedfromthe\nembeddingspacelearnedfromatrainingsetofwords(i.e.thevideotranscripts).Isthiskindofinterpolation\nfundamentallydifferentfrom“memorization”?(Amoreradicalclaimheremightbethatmuchofwhatisseemingly\n“understanding”canbereducedtothiskindofinterpolation—whichIgenerallyagreewith,cf.ourHassonetal.,\n2020paper—butIdoubttheauthorsreallywanttomakethatclaimeither.)\nThisisanimportantpoint,andonealsoraisedbytheotherreviewers:towhatextentarewecapturing\n“understanding”asopposedto“memorization”withourapproach? Asanaside,thisdistinctionis\nactuallynotthemainclaimwearetryingtomakeinourpaper—ourcommentaboutcapturing\nunderstandingversusmemorizationwasintendedasanexampleofhowthesemodelsmightbeuseful\n(toeducators,andsoon). Nonetheless,thereareanumberofimportantdetailstounpackhere.\nFirst,weagreethatthereviewer’sexampleofmemorizingcharacterstrings(likealanguagemodel\nmight)isquitedifferentfromwhatwegenerallythinkofas“rotememorization”ineducational\nse ings. Forexample,memorizingatextbooktothepointthatastudentcould“regurgitate”the\ninformationonanexamstillrequiressomebasicsemanticunderstandingofthematerial,asopposed\ntodoinganythinglikelearningco-occurrencestatisticsofthetextbook’swords.\nAsweclarifyonpage3,wearenottryingtoclaimthatthetopicmodels(orevenothermore\nsophisticatedlanguagemodelswecouldhaveusedinstead,likeChatGPT)arethemselves\n“understanding”thematerialinanycolloquialsenseoftheword. (Weviewthosemodelsmorealong\nthelinesofthereviewer’sdescription—essentiallylearningco-occurrencestatisticsofthetext.)\nRather,ourapproachistousetheembeddingspacesthatthesemodelsdefineasawayofconstructing\nmapsofwhatpeoplemightknow(oreventhinkabout). Specifically,theembeddingspacesthese\nmodelslearnhavethepropertythatnearbycoordinates(e.g.,inEuclideandistance)tendtoreflect\nrelatedconcepts. Inthatsense,thespacesthesemodelslearnare“smooth”(differentiableeverywhere)\nrepresentationsthatwecanusetoefficientlydescribe(andinfer)whatpeopleknoworunderstand.\nThepropertythatendsupbeingmostusefulissmoothness. Becausetheembeddingspaces(learned\nbytextembeddingmodels)havethepropertythatnearbycoordinatesreflectsimilarconcepts,we\nreasonthatknowledgeorunderstandingabouttheconceptreflectedbyagivencoordinatemightimply\nknowledgeorunderstandingaboutrelatedconcepts—whichwilltendtobeatnearbycoordinatesin\ntheembeddingspace.\nWeviewthequestionofwhethertheembeddingspacecapturedbyour(orother)textembedding\nmodel(s)isa“useful”representationofsomeone’sknowledgeasanempiricalone. Inotherwords,\nthesemapsare“useful”totheextentthattheycapturesomethingpredictiveorinformativeaboutwhat\npeopleactuallyknow. Weshowthatwecanpredictindividualparticipants’answerstoheld-out\nquestions(Fig.6),sothistellsusthatsomethingaboutwhatthemodel“considers”tobeconceptually\nrelatedisalsopredictiveofhowparticipants’knowledge“spreads”acrossdifferentconcepts. We\ncharacterizethisspreadmoredirectlyinourrevisedmanuscript(Fig.7)byshowingthedistance(in\ntopicspace)overwhichtheknowledgerepresentationswederiveusingourapproachprovide\npredictivepoweroverandabovesimplycomputingtheproportionsofcorrectlyansweredquestions.\nThedeeperquestionofwhethertheknowledgemapsweconstructfromparticipants’quiz\nperformancecapturetrueunderstandingversussomethingclosertomemorizationistrickiertogetat.\nWethinkthiscomesdowntothespecificquestionsthatweaskedparticipants,asopposedto\nsomethingabouttheembeddingspaces. Ifthequestionsthemselvesrequiredeepunderstanding(e.g.,\nansweringthemcorrectlyrequiressomethingbeyondwhatcanbeachievedthroughmemorization\nalone),thenthemapswelearnfromparticipants’responseswillalsoreflectdeepunderstanding. If\nthequestionsthemselvescanbeansweredbymemorization,thenourmapswillalsoreflect\n“memorized”conceptsasopposedtotrue“understanding.”\nThequestionsweaskedparticipantsinourexperimentvarywithrespecttohowmuchwethinkthey\nmightcapturememorizationversusdeeperunderstanding. Forexample,wesuspectthatquestions\nlike“Inthefamousequationa ributedtoAlbertEinstein,E=mc^2,whatdoesthele er‘m’represent?”\ncouldbeansweredbysomethingclosetopurememorization. Butotherquestions,like“Inyourbody,\nthereareatremendousamountofnegatively-chargedelectrons.Yourcomputeralsocontainsahuge\nnumberofnegatively-chargedelectrons.Weknowthatlikechargesrepel,butyouandyourcomputerare\nnotrepelledapart.Whichofthefollowingexplainswhy?”mightrequiredeeperunderstanding. The\n“amountofunderstanding”agivenquestion“requires”toanswerissomewhatsubjective,andmight\nevencomedowntothe“strategy”agivenparticipantisusingtoanswerthatquestionatthatparticular\nmoment. Wehaveaddedanotetothiseffectonpage31ofourrevisedmanuscript.\nFromamethodologicalstandpoint,whatweseeasthemajorcontributionsofourapproachare(1)a\nsystematicmeansofmappingoutandcharacterizingknowledgeinmapsthathaveinfinite(arbitrary)\nnumbersofcoordinates,and(2)waysoffillingthosemapsusingrelativelysmallnumbersofmultiple\nchoicequizquestions. Ourexperimentalfindingthatwecanusethesemapstopredictresponsesto\nheld-outquestionshasseveralpsychologicalimplicationsaswell. Forexample, conceptsthatare\nassignedtonearbycoordinatesbythetextembeddingmodelalsoappeartobe“knowntoasimilar\nextent”(asreflectedbyparticipants’responsestoheld-outquestions;Fig.6). Thissuggeststhat\nparticipantsalsoconceptualizesimilarlythecontentreflectedbynearbyembeddingcoordinates. The\n“spatialsmoothness”ofparticipants’knowledge(asestimatedusingquizperformance)intext\nembeddingspaceisbeingcapturedbytheknowledgemapsweareinferringfromtheirquizresponses\n(Fig.7). Inotherwords,ourstudyshowsthatknowledgeaboutagivenconceptimpliesknowledge\naboutrelatedconcepts,andwealsoshowhowestimatedknowledgefallsoffwithdistanceintext\nembeddingspace.\nToaddresstheseissues,wehaveaddedclarifyingtextthroughoutourrevisedmanuscript. In\naddition,onpage8weclarifyhowweinterpretandusethetextembeddingsfromthetopicmodels,\nandonpage23weclarifywhatweseeasthemajorcontributionsofourapproach.\n2a.Second,Ithinktheauthorsshouldbeveryclearaboutthelimits/scopeoftheirapproach,particularlybasedon\nthedatasetusedherein.Forinstance,intheIntroduction,youuseaspirationallanguageabout“thesumtotalofa\nperson’sknowledge”and“thenearlyinfinitesetofpossiblethingsapersoncouldknow”—buttheactualdata/model\nusedhereisrelativelysmall-scale(andthere’snoexplicitevidenceforscalability).Ican’timaginethatthe\nco-occurrencestatisticsacrossonly<20minutesofvideolecturesareparticularlydenseorrich.Justtoconfirm,the\nmodelisnotpretrainedinanywayanddoesnotincorporateanyconceptualinformationfromoutsidethelecture\nvideos,right?Forexample,theauthorssay(line492)“Conceptually,eachtopicisintendedtogivelargerweightsto\nwordsthataresemanticallyrelated*or*tendtoco-occurinthesamedocuments”(myemphasison“or”).Doesthe\nmodelknowthatwordsaresemanticallyrelatedaboveandbeyondtheco-occurrencestatistics?Myunderstanding\nwasthat“semanticallyrelated”isderivedentirelyfromtheco-occurrencesstatistics.Inanotherexample,yousay\n“Knowledgeestimatesneednotbelimitedtothecontentofthelectures.”Buttheembeddingspaceisstrictlylimited\ntotheco-occurrencestructureextractedfromtheseparticularlectures—right?\nWehavemodifiedourtextthroughoutourpaper(includingthestatementsthereviewermentionedin\ntheintroduction)toclarifywhatweseeasthescopeofourwork. Asthereviewernotes,ourcurrent\nstudydoesnotshowthatwecanmapouteverythingsomeoneknows;rather,wearefocusedonlyona\nsmall“region”ofwhatpeopleknow. Tobeclear,thereareinfinitelymanycoordinatesinthatsmall\nregionofspace(i..e,thespaceiscontinuousanddifferentiableeverywherewithintheconvexhull\nformedbytheunionofalllecturetimepoints’andquizquestions’embeddings). Butofcoursethe\nspaceofconceptswefocusoninourworkdoesnot(nearly)spaneverythingsomeonecouldknow,\nlearn,orthinkaboutsince,asthereviewernotes,wetrainthemodelusingarelativelysmallnumber\nofdocumentsthatare(presumably)aboutarelativelysmallsetof“discretizable”learningobjectives.\nWeconfirmthattheembeddingmodelsarenotpre-trainedondataoutsideofthelecturetranscripts\nandquizquestions. Atahighlevel,therevieweriscorrectthatourtopicmodelingapproach(Latent\nDirichletAllocation,orLDA)isdrivenbyco-occurrencestatistics. However,LDAdoesimposesome\nadditionalstructureontheembeddingsbeyondpureco-occurrencestatistics(unlike,forexample,\nLatentSemanticAnalysis,orLSA,whichisessentiallylikecarryingoutPCAonthedataset’sword\ncountsmatrix). LDAa emptstolearnthethemesor“topics”thatunderlietheobservedtext,andeach\ndocument(observation)iscastasaweightedblendoftopics(whichareinturndefinedas\ndistributionsofweightsoverwordsinthevocabulary). ThekeydifferencebetweenLDAand“pure”\nco-occurrencebasedapproachesisthatLDAcandothingsliketeasingapartdifferentusesofthesame\nword. Forexample,supposeawordlike“bat”appearsinadocument. Intheabsenceofother\ninformation,wecouldguessthatthewordmightrefertoawingedmammal,apieceofsporting\nequipment,orsomethingonecandowitheyelashes. ModelslikeLSAwouldinferthatthemesor\n“factors”relatedtoeachofthesethemeswerepresentinthegivendocument. LDA,bycontrast,uses\notherwordsinthedocumenttoparceloutwhichspecificuseof“bat”mostlikelyaccountsforour\nobservation.\nThisabilityofLDAtoidentifylatentthemesenablesthemodeltogeneralize(be erthanmodelslike\nLSA)tonewtexts,aslongasthosenewtextsincludeasetofwordsthatoverlapwiththetrained\nmodel’svocabulary. Inourimplementation,althoughindividualquizquestionsmightusedifferent\nwordstodescribeagivenconceptfromalecture,ourmodelstill“matches”questionswith\nreasonable-seemingsegmentsofvideo,whenwecomparethemodel’sembeddingsofthevideosand\nquestions(Fig.4). Inresponsetooneofthereviewer’srelatedcomments(2c;seebelow),wehavealso\nrunanewanalysisshowingthattheseLDAembeddingsexhibitmoretemporallyspecificmatchesto\nthesestothe\nvideosthanobtainedthroughsimplerwordcountoverlap.\nFinally,thereviewerpointsoutanambiguityinourwordingregardingsemanticrelatednessversus\nco-occurrence. Wehaveclarifiedthatsentencetoread(emphasisadded):“Conceptually,eachtopicis\nintendedtogivelargerweightstowordsthataresemanticallyrelated(asinferredfromtheirtendency\ntoco-occurinthesamedocument).”\n2b.Putanotherway,there’satensioninthewritingbetweenthepotentialgeneralityofthemethodandthetargeted\nnatureoftheembeddingspace(i.e.tunedtothecontentofthetwolectures)usedherein.Here’saspecificexample:In\nmyunderstanding,thenegativecorrelationsbetweenmismatchinglectureandvideotopicweights(line176)arean\nindicatorthattheembeddingspacereflectsan“eitherthisorthat”structurespecifictothetwolectures.Ihavethe\nintuitionthatnear-zerocorrelationherewouldprovidestrongerevidencefor“specificity”(line179)than\nanticorrelation(asweseewiththegeneralphysicsquestions).Iwonderifthistrivializesthedimensionality\nreductionresultsinFigure7—i.e.coulditbeanyotherway?(Maybethisisfine!)\nIngeneral,weagreethatthereissomedegreeof“eitherthisorthat''structuretotheembeddings. For\nexample,whenwelookattheembeddingsofthelecturesandquestions(Figs.3,S2),severaltopic\ndimensionsappeartobe“specific”toFourFundamentalForces(topics2,3,8,12,13,and15),whereas\nothersappeartobespecifictoBirthofStars(topics1,4,5,6,7). Therearealsosometopicsthatappear\nintheaverageembeddingsofbothlecturesandtheirassociatedquestions(e.g.,topics11and14). This\nalsoexplainsthe“negativecorrelations”thereviewerisreferringto. Fromourpriorwork,andthatof\nothergroups(e.g.,Boyd-Graberetal.’s2014chapteron“Careandfeedingoftopicmodels”),this\nfollowsfromafundamentaltradeoffintextembeddingmodels,betweengeneralizabilityand\nspecificity.\nToillustrate,let’sconsidertwoapproachestofi ingagivenmodel. Inbothcases,we’llimagineusing\nthesamemodelarchitecture,thesamenumberofparameters,thesamedatasetsizes,andthesame\ninferenceprocedure. ThedifferenceisthatinhypotheticalapproachA,we’llchooseourtraining\ncorpustospanawidevarietyofcontentfromadiversesetofsources. Bycontrast,inhypothetical\napproachB,we’llchooseourtrainingcorpustocontaindocumentsthatcoverarelativelynarrow\nscope. Atahighlevel,sincebothmodelshadthesamenumberofobservationsandbothmodelsareof\nthesame“complexity”(i.e.,matchingarchitectures,numbersofparameters,goodnessoffittotheir\nrespectivetrainingdatasets,andsoon),thesecouldbeconsidered“equivalent”models. Anotherway\ntothinkaboutthesemodels,though,isaboutthetradeoffseachmakesbetweengeneralizability(i.e.,\nsomethinglikethechancesthemodelwouldbeabletogeneratea“reasonable”embeddingofsome\nrandomlyselectedheld-outtext)andspecificity(i.e.,somethingakintothedegreetowhichthe\nembeddingsareabletoreliablydifferentiatebetweensubtlydifferenttextsorconcepts). Essentially,a\nmodel’sdesign,parameters(e.g.,numberofembeddingdimensions),andtheamountandqualityof\ntrainingdatacollectivelydeterminetheeffective“resolution”ofthemodel–i.e.,relatedtothefidelity\noftheoverallembeddingspace. Butwithinthoseconstraints,thechoiceoftrainingcorpusdetermines\nwhethertheembeddingspacespansawiderangeofcontentatarelativelylowerresolution(asinthe\n“approachA”exampleabove),orwhethertheembeddingspace“zoomsin”onanarrowerrangeof\ncontentatacomparativelyhigherresolution(asinthe“approachB”exampleabove). Inotherwords,\ntheparticularsetofdocumentsweusetotrainanembeddingmodelendsupdeterminingthebreadth\nanddepthoftheembeddingspace.\nOnewaythiscanplayoutisinthesortof“eitherthisorthat”representationsthereviewerisnoticing\ninourmodel,aswellasthetensionthey’reidentifyingbetween“generalityofthemethodversusthe\ntargetednatureoftheembeddingspace.” Sincewetrainedourtextembeddingmodelondocuments\nthatwereprimarily“about”twosimilarbutpartiallyseparablesetsofconcepts,thesubtleconceptual\nvariationswithinavideoareeach“assigned”severaltopicdimensionsthatenablethemodelto\nrepresentthosevariationswithineachindividualvideo. That’stheaspectofthemodel’sfeaturesthat\nlookslike“switching”orlikean“eitherthisorthat”representation. Ifwehadusedagreaterdiversity\nofdocumentsinourtrainingcorpus,thewithin-videoresolutionwouldhavesuffered,andthat\n“switching”behaviorwouldhavebeenmoresubtle.\nMorebroadly,therearerealchallengestobuildingembeddingmodelsthatarebothsufficiently\nhigh-resolutionwithinagiventopicandsufficientlybroadsoastocoverawiderangeoftopics. For\nexample,embeddingsderivedfromevenlargemodernmodelslikeBERT,GPT-{2,3,4},LLaMa,and\nothers,thatarealsotrainedonenormoustextcorpora,endupyieldingsurprisinglypoorresolution\nwithinthecontentspacespannedbyasinglecoursevideo. Forexample,herearesomecomparisons\nbetweenbasicvisualizationsfromourpaperderivedfromourtopicmodelingapproach(LDA)versus\nanexamplelargelanguagemodel(BERT):\nWehighlightthreegeneraldifferencesintheabovefigure. Inthetoprow(“Trajectories”)weshowa\n3Dprojection(usingPCA)ofthecontenttrajectoriesforeachlecture(blueandgreenlines)andeach\nquestion(blue,green,andpurpledots). TheLDAembeddingsexhibitseveraldesirableproperties\ncomparedwiththeBERTembeddings:\n● TheLDAembeddingsofthelecturesandquestionsare“near”eachother–e.g.,theconvexhull\nenclosingthetwolectures’trajectoriesishighlyoverlappingwiththeconvexhullenclosing\nthequestions’embeddings. TheBERTembeddingsofthelecturesandquestionsareinstead\nlargelydistinct.\n● TheLDAembeddingsofthequestionsforeachlectureandthecorrespondinglecture’s\ntrajectoryaresimilar. E.g.,theLDAembeddingsforlecture1questions(bluedots)tendto\nappearclosertothelecture1trajectory(blueline),andtheLDAembeddingsforlecture2\nquestions(greendots)tendtoappearclosertothelecture2trajectory(greenline). TheBERT\nembeddingsdonotshowthisproperty.\nInthemiddlerow(“Per-questionmatches”),wedisplaythecorrelationsbetweeneachquestion’s\nembeddingcoordinateandtheembeddingofeachmomentofthecorrespondinglecture. Again,the\nLDAembeddingsshowseveraldesirablepropertiescomparedwiththeBERTembeddings:\n● Thetimeseriesplotofindividualquestions’correlationsaredifferentfromeachotherwhen\ncomputedusingLDA(e.g.,thetracescanbeclearlyvisuallyseparated),whereasthe\ncorrelationscomputedfromBERTembeddingsofdifferentquestionsalllookverysimilar.\nThistellsusthatLDAiscapturingsomedifferencesincontentbetweenthequestions,\nwhereasBERTisnot.\n● Thetimeseriesplotsofindividualquestions’correlationshaveclear“peaks”whencomputed\nusingLDA,butnotwhencomputedusingBERT. ThistellsusthatLDAiscapturinga\n“match”betweenthecontentofeachquestionandarelativelywell-definedtimewindowof\nthecorrespondinglectures. TheBERTembeddingsappeartoblurtogetherthecontentofthe\nquestionsversusspecificmomentsofeachlecture.\nInthebo omrow(“Cross-lecturecorrelationmatrices”)wecomparethepairwisecorrelationsbetween\nembeddingsoflecturesandquestionswithinversusacrosscontentareas(i.e.,contentcoveredbythe\nindividuallectures,lecture-specificquestions,andbythe“generalphysicsknowledge”questions).\nTheLDAembeddingsshowastrongcontrastbetweensame-contentembeddingsversus\nacross-contentembeddings. Inotherwords,theembeddingsofquestionsaboutthelecture1(“Four\nfundamentalforces”)materialarehighlycorrelatedwiththeembeddingsoflecture1,butnotwiththe\nembeddingsoflecture2,questionsaboutlecture2,orgeneralphysicsknowledgequestions. Weseea\nsimilarpa ernwiththeLDAembeddingsofthelecture2(“Birthofstars”)questions. Incontrast,the\nBERTembeddingsareallhighlycorrelatedwitheachother.\nTakentogether,thesecomparisonsillustratehowLDA(trainedonthespecificcontentinquestion)\nprovidesbothcoverageoftherequisitematerialandspecificityatthelevelofthecontentcoveredby\nindividualquestions. BERT,ontheotherhand,essentiallyassignsbothlecturesandallofthe\nquestions(whichareallbroadlyabout“physics”)intoatinyregionofitsembeddingspace,thereby\nblurringoutmeaningfuldistinctionsbetweendifferentspecificconceptscoveredbythelecturesand\nquestions.\nWenotethatthesearenotcriticismsofBERT(orotherlargelanguagemodelstrainedonlargeand\ndiversecorpora). Rather,ourpointisthatsimplefine-tunedmodelstrainedonarelativelysmallbut\nspecializedcorpuscanoutperformmuchmorecomplicatedmodelstrainedonmuchlargercorpora,\nwhenwearespecificallyinterestedincapturingsubtleconceptualdifferencesatthelevelofasingle\ncourselectureorquestion. Ofcourseifourgoalhadbeentofindamodelthatgeneralizedto\nsimultaneouslycapturingmanydifferentcontentareas,wewouldexpectourapproachtoperform\ncomparativelypoorlyrelativetoBERTorothermuchlargermodels. Wesuggestthatbridgingthe\ntradeoffbetweenhighresolutionwithineachcontentareaversustheabilitytogeneralizeacrossmany\ndifferentcontentareasusingasinglesetofmodelweightswillbeanimportantchallengeforfuture\nworkinthisdomain.\nWehaveaddedadiscussionoftheseissuestopages23–25ofourrevisedmanuscript,andhavealso\naddedthefigureabovetooursupplementarymaterials(Supp.Fig.6).\n2c.Anotherspecificexample:Theauthorswanttomakeastrongclaimthatthemodeldoesn’trelyonsurface-level\nsimilarityandthatthe“specificwords”usedinthequizquestionshave“veryli leoverlap”withthelectures?But\ntheremustbesomeoverlap(i.e.co-occurrence),right?Inmyunderstanding,acompletelydisjointsetofwords,none\nofwhichoccurredinthesevideotranscripts,wouldnotmapontothisembeddingspaceinanymeaningfulway.To\nbeclear,Idon’tthinkthisisreallyaproblemoraweaknessofthemethod!—I’mjusttryingtoarticulatesome\ndiscomfortwhenreading.\nTherevieweriscorrectthatthereoftenissomeoverlapbetweenthespecificwordsusedinthequiz\nquestionsandthecorrespondingpartsofthelecturethatdiscussthe“matching”concepts. Whatwe\nmeant(andhavenowa emptedtoclarifythroughoutourrevisedmanuscript)isthatthetopicmodels\nprovideadditionalcharacterizationoftheconceptualcontentofthelecturesandquestionsoverand\nabovewhatweseeusingsimplewordmatchingalone.\nTocomparetopic-basedmatching(correlationbetweentopicvectors)withword-basedmatching(e.g.,\noverlapbetweenthewordsinaquestionvs.inaslidingwindowofalecture’stranscript),we\nimplementedasimplemeasureofwordoverlapusingJaccardsimilarity. (TheJaccardsimilarity\nbetweentextAandtextBiscomputedasthenumberofuniquewordsintheintersectionofwords\nfromAandB,dividedbythenumberofuniquewordsintheunionofwordsfromAandB.) Tocarry\noutafaircomparisonwithourtopicmodelingapproach,weranthetextthroughthesame\npreprocessingstepsweusedpriortofi ingthetopicmodels,includingremovingstopwordsand\npunctuation,convertingdigitstoword-form,andsoon. Wecanthencomparethetimeseriesof\ncorrelation-basedsimilarities(usingtextembeddings)versusthetimeseriesofword-based(Jaccard)\nsimilaritiesforeachquestionandlecture. Inthefigurebelow,weshowthetimeseriesplotsfortwo\nexamplequestions(rows;questiontextisdisplayedabovethecorrespondingpanels),eachcomputed\nusingeithertopiccorrelations(leftpanels)orJaccardsimilarities(rightpanels):\nThequestioninthetoprow(Question5aboutLecture1)isaboutcomparinggravityandthe\nelectromagneticforce. Whenweusetopiccorrelationstomatchupthequestion’sembeddingwith\nLecture1timepoints(leftcolumn),wecanseeaclearlydefined“peak”wheretheembeddingsshow\nstrongcorrelations. WhenweexaminetheLecture1transcriptduringthatinterval,wecanseethatthe\ninstructorisdescribinghowthosetwoforcesrelate(emphasisadded):\n“...disappearsasanactualforceasanactualinteractionnowthenextthenextforceupthe\nhierarchywhichisonethatwearemorefamiliarwithitissomethingit'swhatactually\ndominatesmostofthechemistrythatwedealwithandelectromagnetismthatwedealwith\nandthat'stheelectromagneticforcewewriteitinmagentaelectromagneticmagneticforceand\njusttogiveasensethisisthisis10tothe36timesthestrengthofgravity10to36timesthe\nstrengthofgravitysoitkindofputstheweakforceinitsplaceit's10tothe12timesstronger\nthantheweakforcesothesearehugenumbersthatwe'retalkingabouteitherthisrelativethat\noreventhisrelativetogravityandsoyoumightbesayingwellyouknowtheelectromagnetic\nforcethat'sunbelievablystrongwhydoesn'tthatapplyoveroveroverthesethesekindofmacro\nscaleslikegravityletmewritetheremacroscalesmacroscaleswhydoesn'titapplytomacro\nscalesandactuallythere'snothingabouttheelectromagneticforcewhyitcan'titoritactually\ndoesapplyoverlargedistancestherealitythoughisyoudon'thavethesehugeconcentrations\nofeitherelectriccoulombchargesormagnetismthewayyoudomasssothemassthatyouhave\nsuchhugeconcentrationsitcanoperateoverhugehugedistanceseventhoughit'swaywayway\nweakerthantheelectromagneticforcetheelectromagneticforcewhathappensisbecauseit's\nbotha ractiveandrepulsiveittendstokindofsortitselfoutsoyoudon'thavethesehuge\nhugehugeconcentrationsofchargenowtheotherthingyoumightbewonderingaboutisyou\nknowwhyisitcalledtheelectromagneticforceinoureverydaylifethere'sthingslikethere's\nthingsliketheCoulombforcethatortheelectrostaticforcewhichwe'refamiliarwithpositive\nchargesorlikechargeswanttorepelifbothofthesewerenegativethesamethingwould\nbe…”\nIncontrast,simplewordmatching(rightcolumn)identifiesmomentswhenanyofthewordsinthe\nquestionareusedduringthelecture(butnotnecessarilyalltogether)--e.g.,mentionsof“following,”\n“difference,”“gravity,”“electromagnetic,”and“force”atothermomentsinthelecturealsoevoke\nincreasesinJaccardsimilarity.\nSimilarly,thequestioninthebo omrow(Question15aboutLecture1)isabouttheCoulombForce.\nAgain,ourtopiccorrelationsapproach(leftcolumn)yieldsawell-definedmatchintervalthat\ncorrespondstoanintervalinthelecturewhentheinstructorisdescribingtheCoulombForce\n(emphasisadded):\n“...electromagneticforceinoureverydaylifethere’sthingslikethere’sthingsliketheCoulomb\nforcethatortheelectrostaticforcewhichwe’refamiliarwithpositivechargesorlikecharges\nwanttorepelifbothofthesewerenegativethesamethingwouldbehappeninganddifferent\nchargesliketoa ractwe’veseenthismultipletimesthisistheCoulombforceorthe\nelectrostaticforceelectrostaticandthenontheothersideofthewordyesyouhavethe\nmagneticpartandmagnetsyouknowyouhaveyou’veplayedwithmagnetsonyourfridge\ntheywhat’swhatyouknowifthey’rethesamesideofthemagnetthey’regoingtorepeleach\notherifthey…”\nOntheotherhand,simplewordmatching(rightcolumn)showsnon-specificmatchesthroughoutthe\nlecturetomomentswhenthetranscriptincludes“coulomb,”“force,”or“refer,”butnotnecessarilyall\ntogether(e.g.,otherusesof“force”willevokeanincreaseinJaccardsimilarity,whetherornotthe\nlectureatthosemomentsisspecificallyreferringtotheCoulombForce).\nWeseeanalogouspa ernsforeachquestion. Althoughthetopiccorrelationsarenotalwaysperfect\n(forexample,ourtopicmodelappearstooccasionallyconfuseforcesthatarestrongwiththeStrong\nForce),usinganexplicitmodeloftopicsthatusesco-occurrencepa ernsinthetexttoidentifylatent\nthemesprovidesouranalyseswithstructurethatenablesustomatchupquestionsandlecture\ntimepointswellinpractice:\nIntheabovefigure,eachdotdenotesasinglecomparisonbetweenalecturetimepoint(top:Four\nFundamentalForces;bo om:BirthofStars)andasinglequestion. Thebluedotsdenotethe\ncorrelationsbetweentopicvectorsandtheorangedotsdenoteJaccardsimilarities. Ineachcase,the\ntopiccorrelationsexhibitaclear“peak”correlationatjustafewwell-definedmomentsinthelectures,\nandmostoftheremainingtimepointshavecorrelationsnear0. TheJaccardsimilarities,however,\noftenexhibitmuchbroaderrangesofvalues,reflectingthatJaccardsimilaritiesyieldmatchesthatare\nlesstemporallyspecific(i.e.,morediffuse)thanthosecomputedusingtopiccorrelations.\nWehaveaddedabriefdiscussionofthebenefitsofusingatopicmodel(versussimplewordmatching)\nonpages25–26,alongwiththreenewsupplementalfigures(combiningelementsofthefigurespasted\nintoourresponseabove;Supp.Figs.3,4,and5)andanewsupplementaltable(Supp.Tab.3)showing\nlecturetextfromthebest-matchingsegmentsofthelectures,foreachquestion.\n3.Howexactlywerethequestionsconstructedforeachlecture?Forexample,Figure2suggeststhatthequestions\nfollowthetrajectoriesofthelectures,atleasttosomeextent;obviouslythisissensibleand“bydesign.”It’sgreat\nthattheauthorsprovidethefullsetofquestions/answers(TableS1)!However,Ithinkit’simportantthatthe\nauthorsdescribetheirthoughtprocessindesigningthequestionsabitmoreexplicitly.Forexample,howexactlydid\ntheyminimize“surface-level”similaritywhileensuringenoughconceptualoverlapthatthequestionsmapontothe\nembeddingspaceinameaningfulway?\nOneofourlab’sundergraduateresearchassistantsworkedalongsideaMastersstudentwhowas\nrotatinginthelabforamonthtodevelopalistofquestions(theseresearchersareacknowledgedin\nthepaperfortheirvaluablecontributionstothework,buttheydidnotmeetthecriteriaforauthorship\ndiscussedwithallteammembersatthestartoftheproject,asdeterminedbyJRM). Theseniorauthor\n(JRM)taskedthepairofresearcherswithcomingupwith“15conceptualquestionsabouteachlecture,\nalongwith9additionalquestionsaboutgeneralphysicsknowledge.” Tohelpbroadenthesetof\nlecture-specificquestions,theresearcherswerefurtherinstructedtoworkthrougheachlecturein\nsmallsegments,identifywhateachsegmentwas“about”conceptually,andthenwriteaquestion\naboutthatconcept. Thegeneralphysicsquestionsweredrawnfromtheresearchers’courseworkalong\nwithinternetsearchesandbrainstormingwiththeprojectteamandothermembersofJRM’slab. The\nfinalsetofquestions(andresponseoptions)wasreviewedandapprovedbyJRMbeforecollectingor\nanalyzingthetextorexperimentaldata.\nWehaveaddedanotetothiseffectonpages30–31ofourrevisedmanuscript. Wehavealsotoned\ndownourlanguagedescribingourquestionpoolasreflecting“surface-level”versus“conceptual”\ndetailswhereappropriate,asweacknowledgethatwedidnothaveanexplicitprocessforoptimizing\nordesigningquestionstobe“conceptual”beyondourownintuitions.\n4.Theauthorscorrelatedthetopicweightsforeachquestionwiththetopicweightsateachtimepointinthelecture,\nandinFigure4showthat“eachquestionappearedtobetemporallyspecific”(line208).Intuitively,thisseems\nreasonable…Butcouldn’tthisresultalsobeinterpretedascreatingtensionwiththeideathatthequestionsaren’t\nsimplykeyinginto“surface-level”similarities?Putanotherway,ifIthoughtthisapproachcapturedsomekindof\nholistic“understanding”,Imighthypothesizethatgoodquestionswouldintegratemultipleconceptsintroduced\nacrossdifferentpartsofthelecture.(Imightbeconflatingtheoverallvalidityoftheapproachforcapturing\n“understanding”withthequalityofthequestionshere.)\nThereviewerraisesanumberofinterestingquestionshereaboutwhetherthequestionsweasked\nparticipantsarespecificallycapturingdeeperunderstandingasopposedtosolelycapturing\nsurface-leveldetails. Aswenoteabove,sincewedon’thaveanexplicitdefinitionofwhatspecifically\ndifferentiates“conceptual”versus“non-conceptual”questions,tosomeextentweenduprelyingon\noursubjectivejudgementsinthesema ers. Oursenseisthat,acrossthesetofquestionsweasked\nparticipants,thereissubstantialvarianceinhow“deeply”eachquestionprobesconceptual\nunderstandingversussomethingmoreakintorotememorization. Wealsodiscussedthisissue,along\nwithsomeexamplequestionsthatweseeasvaryingin“depth,”inourresponsetothereviewer’s\ncomment1. Insummary,weacknowledgethatatleastsomeofourquestionslikelydonotcapture\nwhatwemeanbydeepunderstanding,andforotherquestionsitisdifficulttoobjectivelysayhow\ndeepanunderstandingisrequired. Toaddressthispoint,wehavetoneddownhowwedescribeour\nexperiment’squestionsthroughoutourrevisedmanuscript,andwehavealsoaddedanotetothis\neffectonpage31.\nTheabovesaid,wecansaysomethingsaboutwhatourapproachisversusisn’tcapturing. For\nexample,consideranembeddingspacethatfailstocaptureanydeepersemanticmeaning(e.g.,asina\n“random”embeddingspaceforwhichEuclideanorcorrelation-baseddistancewasfullyuncoupled\nfromsemanticmeaning). Insuchaspace,therewouldbenopredictivevalueoftheembeddings. In\notherwords,knowingthataparticipantansweredagivenquestioncorrectlywouldprovideno\nadditionalinformation(onaverage)abouttheparticipant’sknowledgeaboutthecontentcoveredby\nquestionsthatwerenearbyintheembeddingspaceascomparedtoquestionsthatwerefaraway. On\ntheotherhand,ifanembeddingspacedoescapturesimilaritiesinsemanticmeaninginawaythat\nfollowssimilaritiesinparticipants’knowledge,thenparticipantsshouldhavesimilar“levelsof\nknowledge”aboutquestionswhoseembeddingswerenearbyinthespace.\nOfcourse,knowledgeitselfcannotbedirectlymeasured;itcanonlybeinferred“indirectly”through\nbehaviorsorotherobservations. Buttotheextentthatparticipants’knowledgeaffectstheirchancesof\nansweringquestionscorrectly,theembeddingspaceslearnedbyourtopicmodelingapproach,and\nourframeworkforestimatingwhatwecall“knowledge,”seemtohavesomeexplanatorypower(in\ntermsofpredictingresponsestoheld-outquestions).\nWealsolackanydirectmeansofdefiningordeterminingthe“depth”ofsomeone’sknowledge. We\nfoundthattheembeddingcoordinatesofaheld-outquestion,alongwithparticipants’performanceon\notherquestionsatnearbycoordinates,reflecthowtheparticipantwillperformontheheld-out\nquestion(Fig.6). Wealsofoundthatexplanatorypowerfallsoffsmoothlywithdistanceinthetext\nembeddingspace(Fig.7). Thissuggeststhatwearecapturingsomethingmorelike“actualmeaning”\n(asreflectedinsemanticrelationshipsbetweendifferentquestions),overandaboveexacttextmatches\n(arguablythemostbasicframingof“surfaceleveldetails”).\nWithrespecttothereviewer’spointabouthowconceptsmayintegrateacrossdifferenttimepoints,we\ndonotseetemporalspecificityasnecessarilyintensionwithhow“deep”or“complex”(oreven\n“holistic”)aparticularconceptis. Weseethisasdependingonseveralfactorsincludingwhatthe\nconceptitselfis,theparticularsofthelectureinwhichtheconceptisexplainedordescribed,the\ninstructionstyle,andsoon. Presumablythetemporalspecificityalsodependsonthetextembedding\nmodel’ssensitivitytodistinctaspectsofthesameoverarchingconcept.\nAnotherwayofconsideringthereviewer’scommentabouttemporalspecificityversusdepthmaybe\ncapturedinpartbyourresponsetotheircomment2c,above. Inourresponsetothatcomment,we\nreportonananalysisspecificallyaimedatcomparingthetemporalspecificityofquestion-lecture\nmatchescomputedusingtopicmodelsversussimplermeasuresofwordoverlap. Wefoundthatour\ntopicmodelingapproach,whichinherentlycapturesdeeperstructureandco-occurrencepa ernsthan\nwordoverlapalone,resultsinmoretemporallyspecificity. Thistellsusthat,atleastforthosetwo\nexampleapproaches,greatertemporalspecificityisconsistentwithamodelthatcapturesmore\nsemanticdetail(whichonecouldinterpretasa“deeper”semanticrepresentation,orasreflecting\ngreaternuancesin“similarity”).\n5.Theauthorsusedwindowscomprising30lines(?)ofthetranscripttodefinedocumentsforinputtothetopic\nmodel.Thisseemslikeanimportantparameterasitimpactstheco-occurrencestructureobservedbythetopicmodel,\nright?Howwasthisnumber30chosen?Iassumeitrepresentssomehappymediumbetweenprovidingawide\nenoughwindowtoyieldrichco-occurrencestructurewhileretainingsometemporalspecificity.Transcript“lines”\nseemslikeasomewhatarbitraryunit;IassumeitrelatestohowYouTuberendersthetranscript?Howmanywords\ntendtooccurineachofthesewindows?What’stheaveragedurationofthesewindowsinseconds?\nThereviewerisraisinganumberofimportantquestionsabouthowwechosethe“parameters”ofour\ngeneralapproach. Inourpriorwork(Heusseretal.,2021,NatureHumanBehaviour),wherewe\ndevelopedasimilar“topicmodelingofslidingtextwindows”approach,wewerecomputingtext\nembeddingsofhuman-generatedannotations. Wereasonedinthatworkthateachsentenceofthe\nannotationscouldbetreatedroughlyasa“unitofthought”forthepurposesofouranalyses.\nEssentiallychoosingthelevelofresolutionoftheanalysis(le ers,words,sentences,paragraphs,\nentiredocuments,etc.)determinesthe“resolution”atwhichdifferencesbetweensuccessive\nobservationscanbedetectedorcharacterized. Sentencesseemedlikeareasonablechoiceinthatprior\nwork.\nInourpresentstudy,wetriedtofollowourpriorapproachascloselyaswecould,butdueto\ndifferencesinthetextandrequirementsofourapproach,someadaptationswereneeded. Forexample,\ninourpresentworkweusedautomaticallygenerated(viaspeechtotextalgorithms)transcriptsthat\ndidnotincludepunctuation. Further,KhanAcademylecturevideosareoftennarratedinasortof\n“streamofconsciousness”stylethatoftendoesn’tmapcleanlyontosentencesperse. Instead,we\ntreatedeach“line”oftheautomaticallygeneratedtranscriptasastandinfora“sentence.” Togivea\nsense,eachlineoftheautomaticallygeneratedlecturetranscriptscorrespondstoasingleblockof\nclosedcaptioningtextthatisdisplayedon-screenduringagivenpartofthelecture. Weselecteda\nwindowlengthof30“lines”inourcurrentpapertomatchthesameaveragenumberofwordsper\nslidingwindow(beforepreprocessing,roundedtothenearestword)asinthesentence-based\ntranscriptsweusedinourpriorwork(Heusseretal.,2021). Theslidingwindowshadanaverageof\n73.8wordsandlastedforanaverageof62.22seconds.\nWehaveaddedsomeadditionalexplanationofhowwechosetheslidingwindowlengthsinour\nstudy,alongwiththeaveragewordcountanddurationperwindow,topage33ofourrevised\nmanuscript.\n6.Idon’tfullyunderstandthemotivationforchoosing15topicsformostoftheanalyses,orwhyyouincreaseto100\ntopicsforFigure7.Idon’tthinkthere’sanythingwrongwithpickingasomewhatarbitraryroundnumberinthese\ncases;justcuriousifthere’sanyparticularreasoningbehindthepresentchoices.\nThischoicewasmotivatedbysomeobservationsabouthowUMAPworksandsomechoicesthataffect\nhowaccuratelytheembeddingsofthedatainthelowdimensionalspacereflecttheoriginal\nhigh-dimensionaldata. Togivesomecontext,theUMAPalgorithmiswhatweusetovisualizethe2D\n“maps”byconvertingcorrelationsintheoriginal(high-dimensional)topicspaceontoEuclidean\ndistancesinthe2Dmapspace. Thispropertymakesitpossibletointuitivelyvisualizethesimilarity\nrelations(correlationsbetweentopicvectors)wethinkaremostreflectiveofactualsemantic\nsimilarities. However,in“solving”forthemappingbetweenthehighandlowdimensionalspaces,\nUMAPendsupdistortingthedatainafewdifferentways. Someofthesedistortions,suchas\ninformationlosswhenwemovefromhigh-dimensionaltolow-dimensionalrepresentations,are\nunavoidable. Butothersappeartodependinpartonpropertiesoftheoriginalhigh-dimensionaldata,\nincludingthenumberofdimensions,numberofobservations,etc.,andonthenumberofdimensions\ninthetargetlow-dimensionalspace. Wenotedtwotypesofdistortionsinparticular. First,UMAPhas\natendencyto“snap”nearbypointsintheoriginalhigh-dimensionalfeaturespaceontothesame(or\nverynearby)coordinatesinthelow-dimensionalprojection.Second,becausethelow-dimensional\nembeddingiscomputedusingclustersofnearbyobservationsintheoriginal(high-dimensional)\nspace,observationsinthelow-dimensionalspacecanbe“clumped”sothattheyappearclosertogether\ninthelow-dimensionalprojectionthantheyareintheoriginalspace. Wecanseehowthesetwotypes\nofdistortionsappearinthe2Dprojectionsof15versus100dimensionaltextembeddingsofthesame\nlecturesandquestions:\nWhenwe“connectthedots”betweensuccessiveobservationsinthelectures(leftcolumn),bothsetsof\ntrajectorieslook“reasonable”(e.g.,they’rerelativelysmooth,thetwotrajectoriesarepartially\noverlappingbutmostlydistinct,thequestions“about”eachlecturecorrectlymapontotheappropriate\npartsofthe2Dspace,andsoon). Butwhenweexaminetheembeddingsoftheindividualtimepoints\nofthelectures(rightcolumn),it’sclearthattherearemore“gaps”inthe15-topicversion. Sincewe\nknowthatbothtrajectoriesaresmoothintheoriginalhigh-dimensionalfeaturespace(becausewe\ncomputedthemusingoverlappingslidingwindowsandresampledeachtrajectorytohaveaconstant\nsamplingrate),it’sclearthatthe100-topicversionsmoreaccuratelyreflecthowtheoriginal\nhigh-dimensionaldatabehaves. Wethoughtfewergapswouldalsobedesirablewhenvisualizingthe\n2Dknowledgemaps,whichrequireinterpolatingfromnearbyobservations(sogapsinthedata\nrequiremoreinterpolationandlessrelianceonactualobservations). Toselectanappropriatenumber\noftopics(k)forourmodel,asastartingpoint,weidentifiedtheminimumnumberoftopicsthat\nyieldedatleastone“unused”topic(i.e.,inwhichallwordsinthevocabularywereassigneduniform\nweights)aftertraining.Thissuggestedthatthenumberoftopicsmightbesufficienttocapturetheset\noflatentthemespresentinthetwolectures.Wefoundthisvaluetobek=15topics.Wefoundthat\nwithalimitednumberofadditionaladjustmentsfollowingBoyd-Graberetal.(2014;e.g.,removing\ncorpus-specificstop-words),themodelyielded(subjectively)sensibleandcoherenttopics.\nAlthoughtheabovereasoningsuggeststhatthe100-topicembeddingsarebe ersuitedforcreatingthe\n2Dknowledgemapvisualizations,wealsoidentifiedseveralimportantdifferencesinthetopic\nactivationpa ernswhenweused15vs.100topics:\n● Inthe15-topicmodel,topicactivationspersistforlongerdurationsthaninthe100-topic\nmodel. Wethoughtthattheselongerdurationsmightboostthesensitivityofourapproachby\nintegratingoverlongerspansoftime(e.g.,undertheassumptionthattheestimatedtopic\nweightsare“noisy”).\n● Inthe15-topicmodel,topicactivationsoftenoverlap(e.g.,inagiventimepoint,severaltopics\noftenhavelargeweights),whereasthishappensonlyrarelyinthe100-topicmodel. This\npropertyofthe15-topicmodelseemedpotentiallyusefulforcapturinginterestingco-variation\nacrosstopics.\n● The15-topicmodel(roughly)“segments”thelecturesintoontheorderof5-10chunks\n(roughlyoneperminute). Whenweexaminethetemporalcorrelationmatrix,wecansee\nroughly5-10square“blocks”ofstrongcorrelationsalongthediagonal. The100-topicmodel\nsegmentsthelecturesmuchmorefinely(approachingaround10secondsper“block”).\nIntuitively,wewantedtocapturebroadconceptsthatwethoughtmighttakeontheorderofa\nminuteto“explain,”asopposedtomuchlower-levelconceptsthatmightbementionedina\nfewseconds. Thisagainledustopreferthe15-topicmodelinourmainanalyses.\nThesepa ernsmaybeseeninthefigurebelow(left:15-topicmodel;right:100-topicmodel;firstand\nthirdcolumnsshowtopicactivationsovertime;secondandfourthcolumnsshowthetemporal\ncorrelationmatrices;toprowcorrespondstotheFourfundamentalforceslectureandthebo omrow\ncorrespondstotheBirthofstarslecture):\nWebrieflydiscusssomeofthislogiconpages18–19. Wenotethatallofouranalysesrelyonthe\n15-topicmodel,andwecarryouttherelevantcomparisons,distancecomputations,andstatisticaltests\nintheoriginal15-dimensionalspaceforthatmodel. Weusedthe100-dimensionalmodelsolelyfor\nvisualization,whenwecreatedtheknowledgemaps(Fig.8).\nMinorcomments:\nRegardingmypreviouscommentsonscalability,dotheauthorsthinkthiskindofapproachcouldbeextendedfrom\ntopicmodelstothecontextualembeddingslearnedbylargelanguagemodels?Couldbeworthmentioningifthisisa\nvalidfuturedirection,giventhecurrentpopularityofthesemodels.Myguessisthatifwewantamethodthatcan\ncapture“thesumtotalofa\nperson’sknowledge”and“thenearlyinfinitesetofpossiblethingsapersoncouldknow,”\nlargelanguagemodelsarecurrentlythebestapproach.\nWesharethereviewer’sintuitionthatlargelanguagemodelsmaybethemostpromisingwaytoscale\nupourapproachtoabroaderrangeofconcepts. Aswenotedinourresponsetothereviewer’s\ncomment2babove,simplyswappinginalargelanguagemodeltrainedonamuchlargerandmore\ndiversecorpusdoesnotseemto“justwork.” Rather,aswediscussabove,we’vefoundanoverarching\ntradeoff:whenweincludealargerandmorediversesetofcontentinourtrainingcorpus,orwhenwe\nusedsubstantiallylargermodels(e.g.,BERT),wefoundthattheresultingmodelsaremore\ngeneralizabletobroadrangesofcontent,butlessabletopickuponlow-leveldetails(e.g.,atthelevel\nofindividualquestionsorsmallsegmentsofindividuallectures). We’vebeenstartingtothink\nthroughhowonemightconstructmodelsthatworkwellatavarietyofconceptualscales. Oneidea\nwe’vehadistopotentiallycombinemultiplemodelsintoasingle“hybridmulti-scalemodel.”\nAnotheristore-thinkhowtheembeddingsthemselvesareusedinternallybythemodels,orhowwe\nusethoseembeddingstoconstructmaps. Whilewe’reexcitedaboutthesepossibilities,wealsofeel\nthatthey’rebeyondthescopeofourpresentstudy.\nInFigures2and7,youusePCAtovisualizetwo-andthree-dimensionalprojectionsoftheembeddingspace.What\nproportionofcumulativevariancedothesefirsttwoandthreePCsaccountfor?\nInFigure2C,weusePCAtovisualizethe3Dprojectionsoftheembeddingspace. Thosefirst3\ncomponentsexplain36.2%ofthevarianceintheoriginalembeddingsofthelecturevideosand\nquestions. InFigure7weuseUMAP(notPCA)tovisualizetheembeddingspacesas2Dmaps.\nUnlikewithPCA,there’snostraightforwardwayofestimatingvarianceexplainedbyUMAP,sinceit\nisaformofnonlineardimensionalityreduction. (LelandMcInnes,theleadauthoroftheUMAP\nalgorithm,explainsthisissuefurtherhere:h ps://github.com/lmcinnes/umap/issues/122).\nAllparticipantssawtheFourFundamentalForcesvideofirst,followedbytheBirthofaStarvideo,right?Should\nwebeworriedaboutsomeordereffect?\nAllparticipantsinourstudyviewedthetwovideosinthesameorder. Thereforeit’sofcourse\npossiblethattheremightbeordereffectsthatwearenotaccountingfor(i.e.,wedidnot\ncounterbalancetheviewingorder). Thatsaid,wearenotawareofanyreasonwhyviewingorder\nshouldspecificallyinfluenceoraffectourmainfindings.\nIfthetwolectureshadadependencyrelationship(e.g.,understandingonelecturerequiredfirst\nhavinglearnedthecontentoftheother)thenthismightintroducemoreseriousordereffects. Butwe\nspecificallyselectedthetwolecturesinourstudytobeself-contained(andthereforeconceptually\nindependentofeachother). Wementionthisasoneofthecriteriaforchoosingthesetwoparticular\nvideosonpage6.\nLine283:Theset-valueshave600+degreesoffreedom;Iassumethisisdeterminedbythetotalnumberofquestions\nacrossallparticipants.Shouldwebeworriedthatthiskindofstatisticaltestdoesn’ttakeintoaccountsubject-level\nvariability?\nWegeneratetheknowledgeestimatesforeachheld-outquestionusingonlythesameparticipant’s\nresponsestootherquestionsonthesamequiz(Fig.6). Sothereisno“leakage”acrossparticipantsin\nourpredictions(e.g.,thefactthatotherparticipantstendedtogetaparticularquestionwrongwillnot\ninformourpredictionsofhowthecurrentparticipantansweredthatquestion). Ourstatisticaltestsare\ncarriedoutonthefulldistributionofknowledgeestimates;specifically,wecomparetheestimated\nknowledgeofheld-outquestionsthatwereactuallyansweredcorrectlyvs.held-outquestionsthat\nwereactuallyansweredincorrectly. Giventhatouroverallapproachisalreadyrelativelycomplicated,\nweoptedforasimpleandeasy-to-interpretstatisticaltest,ratherthana emptingtomodel\nparticipant-leveleffectsorotherfactors(e.g.,ofquestionidentity,contentarea,etc.). Whilean\nimperfectsolution,wemadeanincrementalimprovementtoourstatisticalframeworkbyusing\nnon-parametricMann-WhitneyU-testsinsteadoft-tests.\nLine492:Doyouuseaspecificsetofstopwords?(e.g.theNLTKpackagehasastandardlistofstopwords)\nWeusetheNLTKpackage’s“English”stopwordslist,augmentedwiththefollowingadditionallistof\nwords(identifiedfollowingBoyd-Grayberetal.,2014):['actual','actually','also','bit','could','e','even',\n'first','follow','following', 'four','let','like','mc','really','saw','see','seen','thing','things','two']. We\nclarifythispointonpage33.\nLine557:“insure”>“ensure”\nFixed!"}, {"round": 2, "reviewer_comments": {"reviewer_1": "The authors have adequately addressed my comments. I appreciate the very thorough response and I endorse the manuscript for acceptance. The comparisons to BERT and word-level Jaccard similarity, and the accompanying explanations, were also very helpful for understanding what insights their approach provides.", "reviewer_2": "I am happy with the responses and edits from the authors in the updated version of the manuscript. In particular, I find the across-lecture knowledge prediction and the \"smoothness of knowledge\" analyses to be important and interesting, and make a satisfying contribution to the full set of empirical results.\nThere a few new sentences that are hard to parse, along with some typos, so the authors should take care to make those fixes. I don't need to see an updated version, but if the authors find it useful I can give a few examples:\np. 14 - line 339 - \"and and\"\np. 23 - line 470 - \"The 'spatial smoothness'... is being captured by the knowledge maps we are inferring from their quiz questions.\"\np. 36 - line 803 - \"...from our dataset full dataset.\"\nI should also note that new edits on lines 72 and 738 are cut off on the provided pdf and so I cannot speak to their content.", "reviewer_3": "See attached PDF."}, "author_response": "The authors clearly put in a lot of work to address Reviewer 3’s concerns. My judgment of the manuscript remains similar to what it was during the last submission, though I do have some comments relating to the new content/analyses and how it is described and interpreted.\n1. The “predicting success on held-out questions using estimated knowledge” section is the one that went through the largest overhaul. I still find the within-lecture and across-lecture analyses to be important, though it is unfortunate that with the new methods the results are harder to interpret. The authors do a good job of trying to explain the pattern of results (sometimes knowledge estimates do predict quiz question accuracy, and other times they do not), though it is not a strong set of results with clear empirical insights. However, since in my view the value of this work is in its methodological rather than empirical contribution, I do not see this as a reason to not approve the manuscript. That being said, I do have a (a) point of confusion, and (b) question/suggestion regarding the analysis.\na. In the rebuttal document, the authors explain an issue regarding the models incorporating the percent-correct measure, in which using percent-correct alone results in a negative relationship between estimated knowledge and probability of answering a question correctly (rebuttal starting at page #3). Given this strange relationship, the authors say that they have decided not to include these results in their paper. However, in the paper, the authors do include analyses that relate to this negative null result (starting manuscript page #16). I assume that the authors are referring to slightly different analyses in their manuscript vs. rebuttal, but if they decided to include analyses with this negative relationship in the manuscript, why did they leave out other ones? The fact that I am not clear why certain analyses were included or excluded makes me feel like I am missing an important aspect of these analyses, but perhaps this is a fault of my own.\nb. In any event, the situation in which the null result is a negative relationship causes confusions and complications (regarding the results displayed in Fig. 6). If the null result is negative (and not zero), then shouldn’t the significance of the results be assessed relative to that negative value (rather than zero)? For example, might there be some observed relationship that may not itself be significantly positive, but still different from the negative null? If so, and if I understand the author’s analysis correctly, then might they not be using a significance test that is actually too conservative? This might be a situation in which generating a null distribution through permutation testing might be the best solution; then the significance of the observed relationship can be compared to the null distribution to determine significance.\n2. The authors discuss their LDA embedding space in relation to more complex (BERT) and simpler (word-matching) models. I have (a) a comment on manuscript organization, and (b) a question about what this means in practice.\na. It seems strange to take up so much space with the comparisons between LDA and BERT while the BERT results are relegated to the supplementary materials. As a style suggestion, it would seem cleaner/more efficient to either move some of these discussions to the supplementary material alongside those results, or to condense the current material and keep in the discussion.\nb. The authors claim that their LDA embedding space is a “sweet spot” in between LLMs like BERT and simpler word-matching models. How specific is this to the particular lectures and quizzes used in the current study? Might e.g. LLMs work better than LDA for other researchers wanting to use these methods? If this is the case, then how should other researchers using their model determine which kind of embedding space to use? What kind of test can be run to determine the correct level of granularity in the embedding space? The proposed model will not be that useful of a tool if it takes thorough testing to determine the appropriate kind of embedding space before any actual knowledge analysis is done.\n3.In their explanation of their Fig. 6 results, the authors suggest that one possibility is that participants forgot some material: “If this forgetting happens in a relatively “random” way (with respect to spatial distance within the embedding space), then it could explain why some held-out questions about Four Fundamental Forces were answered incorrectly, even if questions at nearby coordinates (i.e., about similar content) were answered correctly.” (p. 18).\na. This seems to contradict their theory/finding regarding the “smoothness” of knowledge space. If forgetting happens randomly relative to the spatial location of knowledge, this evokes a kind of “Swiss cheese” knowledge space with holes randomly scattered about—in this case, knowledge would NOT be smooth, as the authors seem to claim. I can’t reconcile these two claims. Perhaps the smoothness visualized in Fig. 7 is a function of averaging/combining across participants, but then making the general smoothness claim would not be so warranted."}, {"round": 3, "reviewer_comments": {"reviewer_2": "The authors clearly put in a lot of work to address Reviewer 3’s concerns. My judgment of the manuscript remains similar to what it was during the last submission, though I do have some comments relating to the new content/analyses and how it is described and interpreted.\n1. The “predicting success on held-out questions using estimated knowledge” section is the one that went through the largest overhaul. I still find the within-lecture and across-lecture analyses to be important, though it is unfortunate that with the new methods the results are harder to interpret. The authors do a good job of trying to explain the pattern of results (sometimes knowledge estimates do predict quiz question accuracy, and other times they do not), though it is not a strong set of results with clear empirical insights. However, since in my view the value of this work is in its methodological rather than empirical contribution, I do not see this as a reason to not approve the manuscript. That being said, I do have a (a) point of confusion, and (b) question/suggestion regarding the analysis.\na. In the rebuttal document, the authors explain an issue regarding the models incorporating the percent-correct measure, in which using percent-correct alone results in a negative relationship between estimated knowledge and probability of answering a question correctly (rebuttal starting at page #3). Given this strange relationship, the authors say that they have decided not to include these results in their paper. However, in the paper, the authors do include analyses that relate to this negative null result (starting manuscript page #16). I assume that the authors are referring to slightly different analyses in their manuscript vs. rebuttal, but if they decided to include analyses with this negative relationship in the manuscript, why did they leave out other ones? The fact that I am not clear why certain analyses were included or excluded makes me feel like I am missing an important aspect of these analyses, but perhaps this is a fault of my own.\nb. In any event, the situation in which the null result is a negative relationship causes confusions and complications (regarding the results displayed in Fig. 6). If the null result is negative (and not zero), then shouldn’t the significance of the results be assessed relative to that negative value (rather than zero)? For example, might there be some observed relationship that may not itself be significantly positive, but still different from the negative null? If so, and if I understand the author’s analysis correctly, then might they not be using a significance test that is actually too conservative? This might be a situation in which generating a null distribution through permutation testing might be the best solution; then the significance of the observed relationship can be compared to the null distribution to determine significance.\n2. The authors discuss their LDA embedding space in relation to more complex (BERT) and simpler (word-matching) models. I have (a) a comment on manuscript organization, and (b) a question about what this means in practice.\na. It seems strange to take up so much space with the comparisons between LDA and BERT while the BERT results are relegated to the supplementary materials. As a style suggestion, it would seem cleaner/more efficient to either move some of these discussions to the supplementary material alongside those results, or to condense the current material and keep in the discussion.\nb. The authors claim that their LDA embedding space is a “sweet spot” in between LLMs like BERT and simpler word-matching models. How specific is this to the particular lectures and quizzes used in the current study? Might e.g. LLMs work better than LDA for other researchers wanting to use these methods? If this is the case, then how should other researchers using their model determine which kind of embedding space to use? What kind of test can be run to determine the correct level of granularity in the embedding space? The proposed model will not be that useful of a tool if it takes thorough testing to determine the appropriate kind of embedding space before any actual knowledge analysis is done.\n3.In their explanation of their Fig. 6 results, the authors suggest that one possibility is that participants forgot some material: “If this forgetting happens in a relatively “random” way (with respect to spatial distance within the embedding space), then it could explain why some held-out questions about Four Fundamental Forces were answered incorrectly, even if questions at nearby coordinates (i.e., about similar content) were answered correctly.” (p. 18).\na. This seems to contradict their theory/finding regarding the “smoothness” of knowledge space. If forgetting happens randomly relative to the spatial location of knowledge, this evokes a kind of “Swiss cheese” knowledge space with holes randomly scattered about—in this case, knowledge would NOT be smooth, as the authors seem to claim. I can’t reconcile these two claims. Perhaps the smoothness visualized in Fig. 7 is a function of averaging/combining across participants, but then making the general smoothness claim would not be so warranted.", "reviewer_4": "The manuscript by Fitzpatrick and colleagues describes an application of text embedding models to measure and “map” conceptual knowledge during and after learning about some topics in physics (watching videos and then taking quizzes). I did not review prior versions of this manuscript but have been asked to assess the responses of the authors to prior rounds of feedback. Given this context, below I provide my own evaluation of the manuscript (mostly focused on the responses to Reviewer 3’s concerns), but I do not specifically suggest or request any further edits to the manuscript.\nFirst, it is clear that the authors have put a tremendous amount of work into responding to prior comments from reviewers. I think they have been about as responsive as possible and the paper is very well written. In many respects it is a beautiful paper. Reviewers 1 and 2 seem generally satisfied with the revisions. Reviewer 3, however, raised a number of concerns related to details of the statistical analyses, the specific models used, and the overall significance of the findings. My impression is that the authors have satisfactorily addressed Reviewer 3’s comments related to the statistical analyses and have appropriately contextualized the choice of model (with helpful comments about other models). However, I do share some of Reviewer 3’s concerns about the significance of the work—and these concerns are, I think, less addressable unless the scope of the paper was substantially expanded (which I think is infeasible). Thus, I have a mixed impression: the authors have done some very commendable work on this paper and there is certainly a wow factor to the methods and visualizations (which may be enough for the paper to have impact), but I do have concerns about the actual significance of the work.\nComments\n1. My overarching reaction to the manuscript is that it is more impressive than it is significant. Reviewer 3 shared a similar reaction: ““the authors’ algorithm is cool, but does it sufficiently advance us beyond the capabilities that already exist in large language models, which are available to students and instructors.” Reviewer 3 was perhaps more narrowly focused on the advance of the current methods relative to other large language models or to ChatGPT, but I think the point applies more generally. I might differ from Reviewers 1 and 2, but, to my mind, the empirical results are more of a sanity check than meaningful results on their own. That is, I don’t think these results speak to a theory or debate in the field of cognitive science. Rather, as the authors acknowledge, this is more of a methods paper and I think the empirical results mostly demonstrate that the method is doing what it “should do.” For example, showing that subjects know more about a topic after watching the video or that knowledge about a topic “predicts” performance for similar questions is something that most people would take at face value. Likewise, the fact that questions mapped to fairly specific time points in the lectures is something that almost needs to be true. If the questions are drawn from the lecture content (which they are), then it would be a red flag if the method couldn’t map questions to timepoints in the lecture. Thus, for each of these results, if the model failed to show these effects, it would be a reason to tweak the model, not a meaningful result. (And I think the authors would mostly agree with all of this). In fact, there are quite a lot of degrees of freedom that went into getting results that “make sense.” This is, of course, more evident in the rebuttal than in the actual manuscript.\n2. I think the question is whether the paper does enough, as a methods paper. (Again, I think this is where Reviewer 3’s concerns were, as well). As I note above, I think the authors have done enough on the statistics/analyses front to make a compelling case that the method “works.” That said, the paper does not introduce a new large language model, nor does it even use the latest models. Thus, I think the value as a methods paper comes down to the specific way in which existing models are leveraged to ‘map’ knowledge. Ultimately, I am not particularly convinced that there is much practical value in this method. The authors motivate the ideas in terms of teachers that might want to map what students know. Putting aside concerns about the actual feasibility of a teacher using this, the immediate question that came to my mind was: what is the advantage (gain) of this method relative to much simpler approaches. The authors compare their method against a strawman of a single test grade (e.g., 85%). The argument is that a single test grade makes it impossible to assess which specific topics or concepts a student has learned. That’s true, of course. But, for most of us that teach classes, it is very common (and almost a necessity) that we structure exams so that questions are organized around sub-topics. For example, we might deliberately include a certain number of questions per subject area. It is therefore quite trivial to have sub-scores by topic area. While I genuinely loved the visualizations that were presented in the current paper (again, it is beautiful work), I really struggled to see how or whether these visualizations provide more useful information than sub-scores on an exam. Even for the analyses that seek to predict performance on held-out questions, I was left wondering whether this method is any more sensitive than simple conditional probabilities based on sub-scores. For example, if a student answered the first 4 questions on a sub-topic correctly, what is the probability of answering question 5 correctly? What if they only answered 2 of the first 4 questions correctly? For a methods paper, I think that there is somewhat or a burden to demonstrate that the new method being presented outperforms existing methods. And, ultimately, I was not convinced that the current paper does this. Yes, the high resolution coordinate spaces are impressive and they certainly give the impression of being much more nuanced than a sub-score, but it is harder to see the actual application of that. Can higher-resolution feedback be provided to a student or teacher? And would this higher-resolution feedback make a meaningful difference to learning outcomes? In these respects, there is no clear evidence or argument that this new method is superior to much, much simpler alternatives.\n3. One way to demonstrate the value of the method would be to actually run a classroom experiment showing that students and/or teachers benefitted from the high-resolution maps compared to some simpler alternative. I realize the authors would argue that this is beyond the scope of the current paper (and I agree), but I think if the scope of the current paper is simply to demonstrate the method without demonstrating the actual value of the method, then it is necessarily a less significant contribution.\n4. Relatedly, Reviewer 3 raises the point that it’s not clear the current model would outperform ChatGPT in the core function of assessing conceptual knowledge. I tend to agree. I do get the authors’ points that their method can track and map learning in a way that ChatGPT isn’t capable of (I agree with this), but, to me, it really comes down to the potential value of the method. This gets back to my question of whether the high resolution maps can actually be translated to some meaningful feedback or intervention. But if the goal is simply to have a user-friendly, powerful method for assessing conceptual knowledge, I think ChatGPT is probably the more appealing option."}, "author_response": "The authors clearly put in a lot of work to address Reviewer 3’s concerns. My judgment of the manuscript remains similar to what it was during the last submission, though I do have some comments relating to the new content/analyses and how it is described and interpreted.\nWe thank the reviewer for their positive assessment of our work, and for taking the time to revisit our manuscript to provide further feedback.\n1. The “predicting success on held-out questions using estimated knowledge” section is the one that went through the largest overhaul. I still find the within-lecture and across-lecture analyses to be important, though it is unfortunate that with the new methods the results are harder to interpret. The authors do a good job of trying to explain the pattern of results (sometimes knowledge estimates do predict quiz question accuracy, and other times they do not), though it is not a strong set of results with clear empirical insights. However, since in my view the value of this work is in its methodological rather than empirical contribution, I do not see this as a reason to not approve the manuscript. That being said, I do have a (a) point of confusion, and (b) question/suggestion regarding the analysis.\nIn general, we view the main contribution of our work similarly to the reviewer (i.e., as primarily methodological, rather than primarily empirical). That said, we appreciate and agree that in the previous version of our manuscript, the results of these analyses were somewhat challenging to interpret. This was due to the effects of two separate issues with our analyses. The first was a statistical bias we noted in our previous response letter (and describe in greater detail in our response to comment 1a) that led to an inverse relationship under our null hypothesis between estimated knowledge for a quiz question and the probability of answering it correctly. The second was a conceptual issue (described in our response to comment 1b) that we identified in the course of the current revision with how we had constructed the “null” models used in our significance tests. As detailed in our response to comment 1b, we have now addressed both of these issues and report an updated set of results in our revised manuscript (pp. 14–19). We believe that our updated results are substantially easier to interpret and provide much clearer empirical insights into whether, when, and to what extent our knowledge estimates predict participants’ success on individual quiz questions.\na. In the rebuttal document, the authors explain an issue regarding the models incorporating the percent-correct measure, in which using percent-correct alone results in a negative relationship between estimated knowledge and probability of answering a question correctly (rebuttal starting at page #3). Given this strange relationship, the authors say that they have decided not to include these results in their paper. However, in the paper, the authors do include analyses that relate to this negative null result (starting manuscript page #16). I assume that the authors are referring to slightly different analyses in their manuscript vs. rebuttal, but if they decided to include analyses with this negative relationship in the manuscript, why did they leave out other ones? The fact that I am not clear why certain analyses were included or excluded makes me feel like I am missing an important aspect of these analyses, but perhaps this is a fault of my own.\nWe apologize for the confusion. To summarize, Reviewer 3 had proposed two separate changes to our manuscript, one of which we adopted, and one we did not. Both changes involved using generalized linear mixed models (GLMMs) to assess the predictive power of our knowledge estimates for held-out questions, and both were affected by an issue involving a “negative relationship” in the data to which these models were fit. Specifically, holding out individual questions from a participant’s quiz responses introduces an inverse (or “negative”) relationship between their success on a given held-out question and their proportion-correct score on the remaining (“held-in”) questions. The “issue” with this relationship is not that it is negative per se, but that (as we explain below) it is fundamentally not a relationship that can be estimated numerically, and therefore incorporating it into a model of participants’ success on held-out questions (i.e., by including their remaining-proportion-correct scores as a predictor) leads to intractable issues with the model’s estimation.\nThe change we declined to adopt was the addition of a new analysis that entailed fitting GLMMs with fixed effects for participants’ remaining-proportion-correct scores in addition to their estimated knowledge. As described below, we identified a number of issues with this approach, one of which is that these models cannot be accurately estimated. The other change Reviewer 3 proposed (and which we did adopt) was replacing the Mann-Whitney U-tests previously reported in Figure 6 with analogous tests based on GLMMs. While the models we fit for these tests didn’t directly consider participants’ remaining-proportion-correct scores (only their estimated knowledge), under our null hypothesis, our knowledge estimates reduce to remaining-proportion-correct scores and exhibit the same problematic negative relationship with success on held-out questions. (This is the “negative null” we described in our manuscript when introducing this set of results.) In this case, our inability to accurately model this “negative null” relationship led to a conservative bias in the significance tests presented in our previous submission, which we have now corrected for in our revised manuscript.\nWe describe this “negative relationship” issue and its effects on both proposed analyses in greater detail below. We describe our approach to correcting for it (in the analyses shown in Figure 6) in our response to comment 1b.\nFirst, suppose a participant correctly answers n out of q questions on a given quiz. If we hold out a correctly answered question from their quiz responses, their proportion-correct score on the remaining questions will be (n – 1)/(q – 1), whereas holding out an incorrectly answered question would yield a remaining-proportion-correct score of n/(q – 1). In this way, any given participant’s remaining-proportion-correct score will always be lower for held-out questions they answered correctly than for those they answered incorrectly—in other words, within participants, remaining-proportion-correct is inversely (or “negatively”) related to success on held-out questions.\nIn both of the GLMM-based analyses Reviewer 3 proposed, they requested we fit models with random effects for participant and question identities to account for potential groupings by those factors in our data. Fitting these models therefore entails (in part) estimating any fixed effects they include based on each individual participant’s (and question’s) observations, separately. For the analysis we declined to adopt, which involved fitting models with fixed effects for remaining-proportion-correct, this would include estimating the “negative” within-participant relationship described above. However, this isn’t a statistical relationship we can estimate through regression, but rather it is simply a mathematical byproduct: each participant’s remaining-proportion-correct score can take on only two possible values (n/(q – 1) or (n – 1)/(q – 1), for their particular n), and which of those values it takes on for a given observation is fully determined by the value of the response variable we’re trying to predict (i.e., their success on the held-out question). In other words, this relationship is perfectly deterministic and lacks any residual variance that would allow for statistical estimation. Attempting to fit a logistic model to this relationship results in “complete separation” in the response variable, as the combination of participants’ identities and their remaining-proportion-correct scores perfectly distinguishes between correctly and incorrectly answered held-out questions. If both of these terms were modeled as fixed-effect predictors (e.g., as in a standard logistic regression), this would lead to infinite parameter estimates, making the likelihood function non-identifiable and causing the model to fail to converge altogether. Modeling participants as random effects masks this problem to a degree, as partial pooling across random-effect levels shrinks these estimates towards zero such that it is possible for the model to converge (hence our ability to report a result from this analysis for illustrative purposes in our previous response letter). However, this “shrinkage” does not “fix” the underlying circularity in the model’s formulation, nor the complete separation it creates. Instead, the model the reviewer had proposed would produce unstable and misleading results, as it assigns extreme values to these otherwise-infinite parameters (and artificially shrinks or inflates other parameter estimates to compensate) in an attempt to accommodate a “relationship” that is really a tautology in disguise.\nThese issues also affected the GLMM-based analyses we adopted for Figure 6, but in a slightly less direct way. For each of these analyses (i.e., each panel of Fig. 6), we performed a likelihood-ratio test comparing two different GLMMs: (1) a “full” model we fit to explain participants’ success on held-out questions given their estimated knowledge at those questions’ embedding coordinates, with random effects for participants and questions, and (2) a “null” model we fit to explain the same data given only the full model’s random effects. Formally, these tests assess whether the likelihood of our observed data is significantly greater under the full model than under the null model—in other words, whether a model that considers participants’ estimated knowledge explains their success on held-out questions significantly better than one that does not (or equivalently, whether knowledge estimates provide significant explanatory power). However, because the models we’re comparing differ by only a single fixed effect, these tests are also (asymptotically) equivalent to testing whether the relationship between estimated knowledge and (the log-odds of) success on held-out questions is significantly different from zero—which does not necessarily reflect the expected relationship under our null hypothesis. This is because our estimates of participants’ knowledge for held-out questions are simply weighted versions of their remaining-proportion-correct scores, where the weight we assign to each held-in question reflects its embedding-space distance from the held-out question (with smaller distances corresponding to larger weights).\nOur core assumption in constructing these estimates—which we are testing in these analyses—is that a participant’s ability answer a particular question explains more about their ability to answer other questions at nearby embedding coordinates than about their ability to answer questions at far-away coordinates (i.e., that knowledge of the concepts tested by these questions is “smooth” with respect to distance in embedding space). If this is not the case, as we assume under our null hypothesis (i.e., if the between-question distances we leverage as weights are randomly distributed with respect to participants’ correct vs. incorrect responses), then on average, our estimates of participants’ knowledge will approach their unweighted remaining-proportion-correct scores and exhibit the same inverse relationship with their success on held-out questions. Note that this applies specifically to the “All questions” and “Within-lecture” versions of these analyses (top row and middle two rows of Fig. 6), wherein we estimate knowledge for each held-out question using the remaining questions from the same pool as the one we held out (i.e., all remaining questions from the same quiz, or all remaining questions about the same lecture, from the same quiz). By contrast, in the “Across-lecture” analyses (Fig. 6, bottom two rows), we estimate knowledge for each question about one lecture using only questions about the other lecture. In this case, “holding out” a particular question does not alter a participant’s proportion-correct score on the questions we use to estimate their knowledge.\nThe above means that our significance tests for the “All questions” and “Within-lecture” analyses were overly conservative, because they assumed a “more positive” null relationship (i.e., zero) than we would actually expect given truly uninformative knowledge estimates. Any predictive power our knowledge estimates do afford would therefore have to surpass a relatively high threshold to overcome this \"negative baseline\" and yield an apparent positive relationship. Although assessing our results against the true “negative null” (as the reviewer suggests in comment 1b) would in theory have been a less conservative test, it was unfortunately not clear to us how such a comparison could be operationalized since this null relationship (between unweighted remaining-proportion-correct scores and success on held-out questions) is not a “real” statistical relationship we can represent numerically in order to test hypotheses against. At the same time, we felt that Reviewer 3’s proposed GLMM-based approach was worth adopting for these analyses, as we agreed with their assessment that it accounted for important effects in our data that our prior U-tests had not (e.g., baseline differences in participants’ performance and questions’ difficulties). In weighing these factors, we ultimately concluded that a highly conservative test would be preferable to a potentially anti-conservative one, and opted to include these analyses in our previous submission along with a brief explanation of why an apparent negative relationship actually reflects a null result. However, we acknowledge this was not an ideal solution. As we describe in response to comment 1b, in our newly revised manuscript, we have now devised a correction for this conservative bias that does not require explicitly modeling this negative null relationship and instead addresses the underlying problem that led to it by ensuring that participants’ remaining-proportion-correct scores do not vary with their success on held-out questions (as was already the case in the “Across-lecture” analyses). We have also added text to our revised Methods section describing this “negative relationship” issue and motivating our approach to correcting for it (pp. 36–38).\nFinally, we note that despite having now corrected for this issue, we have still elected not to adopt the additional GLMM-based analysis Reviewer 3 had proposed, as we feel there are remaining problems with its formulation. For this analysis, Reviewer 3 had requested we use a likelihood-ratio test to compare two models (both fit with the appropriate random effects): one with fixed effects for both remaining-proportion-correct and estimated knowledge, and one with a fixed effect for remaining-proportion-correct but not for estimated knowledge. The goal of this proposed analysis was to “provide a direct evaluation of the predictivity of ‘estimated knowledge’ against a baseline”—i.e., to isolate the predictive information uniquely contributed by our knowledge estimates from that already afforded by traditional proportion-correct scores. We agree that this is an appropriate baseline against which to evaluate our knowledge estimates, but do not believe that including these scores as a separate predictor is the right way to accomplish this. Rather, a likelihood-ratio test comparing these two models would assess whether considering participants’ estimated knowledge in addition to their proportion-correct scores better explains their success on held-out questions than considering their proportion-correct scores alone. In other words, this analysis treats estimated knowledge as a measure intended to supplement traditional proportion-correct scores, whereas we propose estimating a learner’s knowledge as an alternative to computing their simple proportion-correct score. Since knowledge estimates are simply weighted proportion-correct scores, these two measures contain partially redundant “information” (and in fact are perfectly collinear under our null hypothesis). Instead, the aspect of our knowledge estimates that can be thought of as “supplementing” traditional proportion-correct scores is the particular weights we use in constructing them. Thus an analysis that achieves the intended goal of this proposed analysis would be one that isolates the predictive information uniquely contributed by these weights from that of the unweighted proportion-correct scores we apply them to. Since our knowledge estimates reduce to unweighted proportion-correct scores under our null hypothesis, and the bias correction we now employ ensures these scores do not vary with a participant’s success on held-out questions, this is exactly what is now accomplished by the likelihood-ratio tests we report in Figure 6. In fact, if we were to apply the same bias correction to this proposed analyses, the estimated effect of remaining-proportion-correct would be exactly zero, since there would be no within-participant variation for it to explain, and its between-participant variation would be the same information already captured by the model’s per-participant random intercepts—making this analysis mathematically equivalent to the analyses shown in Figure 6. In our view, this reflects what information an individual’s proportion-correct score fundamentally can provide about their ability to answer a particular quiz question (that is, none beyond their baseline probability of correctly answering any question), and why the question-specific knowledge estimates we compute therefore yield “higher-resolution” insights.\nb. In any event, the situation in which the null result is a negative relationship causes confusions and complications (regarding the results displayed in Fig. 6). If the null result is negative (and not zero), then shouldn’t the significance of the results be assessed relative to that negative value (rather than zero)? For example, might there be some observed relationship that may not itself be significantly positive, but still different from the negative null? If so, and if I understand the author’s analysis correctly, then might they not be using a significance test that is actually too conservative? This might be a situation in which generating a null distribution through permutation testing might be the best solution; then the significance of the observed relationship can be compared to the null distribution to determine significance.\nIn brief, the reviewer is largely correct in their assessment. This “negative null” phenomenon did indeed render our significance tests highly conservative (specifically for the “All questions” and “Within-lecture” analyses) since these tests effectively assume a null relationship of zero, rather than the “negative” one we would expect. Unfortunately, since this negative null actually reflects a mathematical dependency rather than a “real” statistical relationship with a finite magnitude we can quantify numerically, any typical approaches we might take to constructing a null hypothesis around it would either fail or not be statistically valid. For example, permutation testing would not work here because this “negative null” arises from the direct correspondence between a specific participant’s success on a specific held-out question and their remaining-proportion-correct score excluding that question. Any permutation of the data that breaks this correspondence would therefore no longer exhibit the negative relationship it was intended to capture. In other words, our observations are inherently not exchangeable under our null hypothesis because this negative relationship is not one that could have occurred by chance (and thus could be approximated through random permutation), but rather one that exists due to the circular dependency within each observation.\nWe also considered a number of other methods of accounting for this negative null relationship in fitting these models (e.g., adjusting for it with a global offset term, using GLMMs that incorporate Bayesian priors, residualizing estimated knowledge against remaining-proportion-correct, permuting the questions’ embedding coordinates rather than correct/incorrect labels, etc.) and ultimately came to a similar conclusion as above in each case. We therefore devised a correction procedure that instead works by eliminating this dependency from our data prior to fitting the models. Specifically, this correction ensures that the knowledge estimates we compute for questions a given participant answered correctly and incorrectly are based on the same underlying proportion of correctly answered questions, and therefore have the same expected value under our null hypothesis. In our revised manuscript, we refer to this as a “rebalancing procedure” to mirror the term used to describe analogous approaches to correcting for an analogous problem in leave-one-out cross-validation (e.g., Austin et al., 2024: https://arxiv.org/abs/2406.01652).\nTo summarize, suppose again that a participant correctly answers n out of q questions on a given quiz: their remaining-proportion-correct score will be (n – 1)/(q – 1) when we hold out a correctly answered question and n/(q – 1) when we hold out an incorrectly answered question. To correct for this difference, when we hold out a given question, we identify from the set of q – 1 remaining questions all those with the opposite “correctness” label of the held-out question (i.e., when holding out a correctly answered question, we identify all remaining incorrectly answered questions, and vice versa). We then additionally exclude each of these opposite-label questions, in turn, and estimate the participant’s knowledge for the held-out question using the remaining q – 2 questions. This yields a set of knowledge estimates for the held-out question that are each derived from an underlying proportion-correct score of (n – 1)/(q – 2), regardless of whether the held-out question was answered correctly or incorrectly. We then average over these estimates to obtain a “rebalanced” estimate of the participant’s knowledge for the held-out question. This rebalanced estimate also has an expected value of (n – 1)/(q – 2) under our null hypothesis (since it’s an average over estimates with that expected null value), but effectively “spreads” the shift in its underlying remaining-proportion-correct score equally across all opposite-label questions’ contributions.\nIn our revised manuscript, we use these rebalanced knowledge estimates to fit GLMMs for the “All questions” and “Within-lecture” analyses shown in Figure 6. Importantly, this means that the null-hypothesized relationship between estimated knowledge and the log-odds of success on a held-out question is now zero (as was already the case in the “Across-lecture” analyses), and is therefore accurately reflected by the null models used in our likelihood-ratio tests. This also means that any non-zero relationship we observe in these analyses now reflects predictive information specifically afforded by the embedding-space distances we use to weight individual questions in constructing our knowledge estimates. Finally, in developing this rebalancing procedure, we identified a separate issue with how we had constructed the “null” models used in these tests. Previously, in instances where the data supported estimating group-specific deviations in the effect of estimated knowledge (i.e., fitting a “full” model with “random slopes” for participants and/or questions), we had excluded these random slope terms from our null models in addition to the fixed effect. This is common practice when performing likelihood-ratio tests for the purpose of variable selection, where the goal is typically to determine whether considering a variable “in any capacity” improves the model’s fit to the data. However, since we are using likelihood-ratio tests for the purpose of hypothesis testing, this effectively conflated tests for two different null hypotheses: that there is no relationship between estimated knowledge and the log-odds of success on held-out questions, and that this relationship does not vary among participants and/or questions. In other words, removing these random slopes from our null models actually made our prior likelihood-ratio tests somewhat anti-conservative, since an observed significant result could potentially reflect explanatory power contributed by multiple different terms. This was far overshadowed by the more substantial conservative bias in the “All questions” and “Within-lecture” analyses, but led to false-positive results in the “Across-lecture” analyses for Quiz 1. In our revised manuscript, we have now remedied this issue by retaining these random slope terms in our null models when they are also present in the corresponding full models.\n2. The authors discuss their LDA embedding space in relation to more complex (BERT) and simpler (word-matching) models. I have (a) a comment on manuscript organization, and (b) a question about what this means in practice.\na. It seems strange to take up so much space with the comparisons between LDA and BERT while the BERT results are relegated to the supplementary materials. As a style suggestion, it would seem cleaner/more efficient to either move some of these discussions to the supplementary material alongside those results, or to condense the current material and keep in the discussion.\nWe appreciate this suggestion. We agree it feels odd to devote a large portion of our Discussion section to these comparisons, since (while interesting and useful) they are somewhat tangential to our main results and overall narrative. At the same time, given that both of these comparisons (between LDA and BERT, and LDA and simple word-matching) were motivated by similar questions from multiple reviewers, we suspect that some discussion of their implications will also be of interest to readers more broadly. We have therefore elected to move the bulk of text concerning these comparisons to our Supplementary Information file while retaining a higher-level overview of what we see as their most relevant implications in our Discussion. In our supplement, we have added and expanded on the text related to lower-level comparisons between specific elements of these figures in a new section titled Supplementary results (beginning on p. 22 of our Supplementary Information file).\nb. The authors claim that their LDA embedding space is a “sweet spot” in between LLMs like BERT and simpler word-matching models. How specific is this to the particular lectures and quizzes used in the current study? Might e.g. LLMs work better than LDA for other researchers wanting to use these methods? If this is the case, then how should other researchers using their model determine which kind of embedding space to use? What kind of test can be run to determine the correct level of granularity in the embedding space? The proposed model will not be that useful of a tool if it takes thorough testing to determine the appropriate kind of embedding space before any actual knowledge analysis is done.\nThe reviewer poses a number of interesting questions here. We will attempt to address each in turn.\nThe authors claim that their LDA embedding space is a “sweet spot” in between LLMs like BERT and simpler word-matching models. How specific is this to the particular lectures and quizzes used in the current study?\nTo clarify, it is not our intention to argue that LDA is universally “better” or “worse” than BERT or other large language models, nor do we intend our manuscript to make any claims or comparisons regarding the general usefulness of different model algorithms or architectures. Rather, our point is that if and when our goal is to obtain text embeddings that distinguish between highly similar content within a relatively narrow conceptual domain (e.g., nearby moments in a brief physics lecture, as in our current paper), we suggest that it is beneficial to use a text embedding model trained specifically on the content we want to be able to “explain,” rather than one that has been trained to explain a far broader and more diverse range of content (like BERT and other LLMs).\nThis reflects a fundamental tradeoff in constructing text embedding models (and language models more broadly), between “generality” and “specificity.” One key factor that determines where a particular model falls along the spectrum of this tradeoff is the breadth of conceptual content included in its training corpus. Essentially, the enormous and highly diverse text corpora used to train modern LLMs (in the case of BERT, roughly 3.3 billion words across thousands of books and all of English Wikipedia) enables them to represent content from a vast array of conceptual domains in a way that models trained on smaller, domain-specific corpora (like our LDA model) fundamentally cannot. However, since these models’ embedding spaces must be able to represent any content within the enormous scope of their training materials, this necessarily limits the level of specificity (or “resolution”) with which they can characterize subtle differences between content within a more narrow scope. In other words, while a model trained exclusively on content from a given domain can dedicate its entire feature space to capturing subtle similarities and distinctions that are meaningful within that domain (at the cost of generalizing poorly beyond that domain), a general-purpose model must apportion its representational capacity across a much broader conceptual landscape.\nOne way to think about this is to substitute the semantic features we want to capture using text embeddings for physical features we might instead want to capture through photographs. Suppose that rather than relating and distinguishing lectures and quiz questions based on their conceptual (semantic) features, we wanted to (either qualitatively or algorithmically) relate and distinguish different individuals based on their facial features. If we used photographs containing only those individuals’ faces, we might be able to do this fairly well. But if we were to “zoom out” too far when taking these pictures, to the point where the vast majority of their “space” was devoted to representing content other than the subject’s face (e.g., the surrounding environment), and the subject’s face was represented using only a small number of pixels, then the particular features we care about capturing would be blurred and lost. We might still be able to detect that each image contains a face, but we would fail to capture the more subtle and specific features that identify and distinguish different faces.\nThis is what we see as the key take-away from the comparisons between our LDA model, which we trained exclusively on the lectures used in our study, and BERT, whose embedding space must “span” its far larger and more diverse training corpus (Supp. Fig. 10). The embeddings afforded by our LDA model capture the lecture and question content at a “resolution” that enables us to distinguish between questions about broadly similar but subtly (and meaningfully) different content areas, and to “match” individual questions to specific sections of lecture content. By contrast, BERT assigns both lectures and all quiz questions to the same tiny region of its embedding space, indicating that they are all broadly “about” the same domain of physics concepts, but blurring the more subtle distinctions between the different concepts they cover within that domain.\nTo extend this facial-feature analogy, we could also imagine taking pictures that are “zoomed in” too far, to the point where each image captures only a small patch of skin or hair. In this case, we would similarly fail to capture the features relevant to our goal, as we would be considering only fragments of them—for example, a given patch of skin might be part of a nose, ear, chin, or something else, but we lack the surrounding context needed to determine this. This is what we see as the key take-away from our comparisons to simple word-matching “models” (Supp. Fig. 11). If a given question contains the word “force,” that question could be asking about concepts related to the gravitational force, electromagnetic force, strong or weak nuclear forces, or something else. But the distinction between these possibilities is blurred and lost because the abstract concepts discussed in the lectures and referenced by the quiz questions are not defined or distinguishable at the level of individual words. This is the “sweet spot” notion we describe in our manuscript, between simpler and more complex methods we could use to capture and relate the lectures’ and questions’ contents. While larger, more complex LLMs “take too broad a view” in characterizing their semantic contents, simpler word-matching approaches “take too narrow a view,” and as a result, both alternatives fail to capture conceptual similarities and differences at a level of granularity that would allow us to meaningfully estimate knowledge for the different concepts covered within the space of a single course lecture. But by training a model specifically on the course materials for which we want to estimate participants’ knowledge, we are essentially defining our “level of zoom” directly based on the content of interest itself, enabling the model to devote the full extent of its embedding space to characterizing subtleties within that content, so that the relevant conceptual features “come into focus.”\nIn this way, the reviewer’s question of whether this “sweet spot” is specific to the particular course materials used in the current study has two answers. Given that we trained our text embedding model on the specific lectures our study’s participants viewed and answered questions about, the precise embedding space the model identified from that content is, naturally, optimized for characterizing that content, and we would not expect it to be equally useful for characterizing other very different content. However, the ability to identify an analogously useful embedding space defined by the contents of an arbitrary set of course lectures is a central component of our modeling framework. In other words, the distinction we are drawing here is that the “sweet spot” we refer to in our manuscript is not the exact embedding space our model identified for these course lectures, but rather the approach we take to defining an embedding space for any given set of lectures (i.e., training the model on overlapping sliding windows of their transcripts). To reiterate from our previous response letter, we by no means believe LDA is the only model capable of identifying this sort of “sweet spot” embedding space—in fact, if a different “LDA-like” model were found to characterize the lectures’ and questions’ contents with similar (or possibly even greater) fidelity, that model could easily be “dropped in” in lieu of LDA without changing any of the core assumptions of the broader framework we are proposing. However, we suggest that an important characteristic of any alternative model that could be comparably well-suited for this task (i.e., what we mean by “LDA-like” in this context) is that it can be adequately trained on the specific course materials at hand.\nWe have refined the language in our updated Discussion section to more clearly frame the comparisons between our LDA model and BERT in terms of the conceptual breadth of their training corpora (as opposed to these specific models themselves), and to more clearly present this “sweet spot” idea in terms of the different “semantic scales” at which our approach and other alternatives characterize the lectures’ and quiz questions’ contents (pp. 24–26).\nMight e.g. LLMs work better than LDA for other researchers wanting to use these methods? If this is the case, then how should other researchers using their model determine which kind of embedding space to use? What kind of test can be run to determine the correct level of granularity in the embedding space?\nFollowing from our response above, we do not believe that LLMs are likely to “work better” for other researchers interested in adopting our framework, given the misalignment between their “generalist” embedding spaces and the level of specificity needed to successfully identify and distinguish concepts on the scale at which our framework applies (i.e., within individual course lectures).\nThat said, a common way of improving generalist LLMs’ performance on domain-specific tasks is to “fine-tune” them on a domain-specific text corpus. In our previous response letter, we had speculated that a fine-tuned LLM might capture nuances in the lecture and question content comparably well to—or perhaps even better than—our LDA model. In the interest of answering the reviewer’s question here, we decided to explore this possibility further, and in doing so realized there are some additional reasons why LLMs would likely be poorly suited for our particular application that wouldn't be solved through fine-tuning. We feel strongly that that a discussion of these more technical considerations around different model architectures, fine-tuning techniques, linguistic structures, and so on is outside the scope of our current paper and would distract from our main focus of presenting our more general knowledge estimation framework (similarly to how the extended comparisons with BERT felt out of place in our Discussion). However, since the reviewer is specifically asking about the possibility of incorporating LLMs into our framework, we do think these considerations bear mentioning in response. To help illustrate, we fine-tuned BERT using the same text corpus to which we fit our LDA model (i.e., sliding windows of the two lectures’ transcripts) and reassessed its ability to yield “useful” embeddings in the context of our framework (i.e., embeddings that enable us to distinguish lectures and questions about different content areas, and “match” individual questions to specific sections of lecture content). The plots shown below are analogous to those displayed for our LDA model and “base” (i.e., not fine-tuned) BERT in Supplementary Figure 10:\nIn summary, while fine-tuning leads to a marginal improvement compared to “base” BERT, the model’s performance still falls far short of our domain-specific LDA model’s. For example, the lectures and questions related to different content areas are now represented more distinctly than they were prior to fine-tuning (e.g., the range of correlation values in the heatmap above is ~16 times greater than that shown in Supp. Fig. 10), but we still fail to recover the temporally specific “matches” between questions and periods of lecture content afforded by the LDA embeddings (i.e., compare the correlation time series plots above to Figs. 4A & 4B). This highlights an important difference between fine-tuning a generalist LLM and training a domain-specific model from scratch: even after fine-tuning, LLMs continue to allocate a portion of their representational capacities to content outside the tuning corpus. (This is very much by design, with failure to do so termed “catastrophic forgetting.”) This implies that even for two hypothetical models with identical architectures, parameters, inference procedures, and so on, one that is pre-trained on a domain-general corpus and then fine-tuned to a domain-specific corpus will necessarily afford lower “specificity” for the semantic (conceptual) content within that domain than one that was trained on that domain from scratch.\nDespite this, in many cases this retention of domain-general “knowledge” after fine-tuning is a desirable behavior, and in fact a notable advantage to using a fine-tuned generalist model over a domain-specific model. For example, it can greatly benefit performance when the domain of content the model will need to “handle” is either too large to be fully included in a training corpus, or not fully known or available at the time of training (e.g., if our goal was to characterize scientific papers uploaded to arXiv, it would be intractable to train a model on every paper ever uploaded, and impossible to train it on future uploads). But this benefit doesn’t apply in the context of our framework since the content we (or other researchers and practitioners) might want to characterize (i.e., specific course materials for which we want to estimate learners’ knowledge) will necessarily be known and available in advance. And far from being prohibitively time- or resource-intensive, training a domain-specific model on course materials comparable to those used in our study would be fast and easy (e.g., training our LDA model on the lecture transcripts takes less than one second—roughly 1/1000th the time it took to fine-tune BERT to the same data).\nAnother reason this behavior is often beneficial is that it enables LLMs to retain representations of complex linguistic features (e.g., syntactic features) that are difficult to learn from smaller, domain-specific corpora alone. However, in our particular case, this actually ends up being a further detriment. Transformer-based LLMs are almost universally trained on written text, which tends to follow much more rigid grammatical and syntactic conventions than natural speech. Through their architectures and training paradigms, LLMs are designed to leverage this structure in characterizing a given text sample’s meaning. But in our case, the text samples we want to characterize comprise transcripts of content that was originally spoken (i.e., course lectures), and therefore exhibit different linguistic structures than LLMs are optimized to leverage. For example, the transcripts contain various disfluencies, repetitions, and informalities that rarely occur in written text, while simultaneously lacking many of the grammatical elements that usually provide important “guideposts” for transformer models’ attention mechanisms, such as function words, discourse markers, and sentence boundaries (e.g., punctuation). To provide a concrete example, here is an excerpt of the first ~25 seconds of the first lecture video used in our study:\n“what I want to do in this video is give a very high-level overview of the four fundamental forces four fundamental forces of the universe and I’m going to start with gravity I’m going to start with gravity and it might surprise some of you that gravity is actually the weakest of the four fundamental forces that’s surprising because you say wow that’s what keeps us glued not glued but it keeps us from jumping off the planet”\nWhile it might be possible to manually edit the lecture transcripts to exhibit a structure more typical of written text, we view our ability to leverage automatically generated, freely available transcripts (from the YouTube API) as an important part of our framework’s potential for scaling up to larger and more diverse sets of course lectures, and being useful to other researchers without requiring significant upfront effort or investment. Even if the transcripts themselves were structured more like typical written text, our approach to constructing “trajectories” representing the lectures’ dynamic contents (by projecting sliding windows of their transcripts into text embedding space) means that the individual text samples (i.e., sliding windows) the model “sees” often don’t correspond to cleanly delineated sentences. Meanwhile, the quiz questions—which our goal is to “match” to the lecture content to facilitate our knowledge estimates—did originally appear in written form, and therefore exhibit linguistic structures that differ from the lecture transcripts, and are closer to what transformer models “expect to see.” As we note in our Supplementary results section, this likely poses further challenges to “matching” the questions to specific lecture content based on their embedding weights: since BERT’s representations of the semantic features that relate and distinguish content within this domain are highly homogenous, much of the variation in its embeddings instead reflects the lectures’ and questions’ syntactic differences (e.g., see the 3D PCA projections above and in Supp. Fig. 10).\nDespite being an arguably “less powerful” model than modern LLMs, LDA is actually fairly robust to these challenges that the lecture transcripts’ and quiz questions’ linguistic structures pose for transformer models. Like many comparatively simple text embedding models, LDA makes the simplifying assumption that the text samples it encounters are equivalent to “bags of words.” In other words, it ignores word order and syntax entirely in characterizing a text sample’s semantic meaning and instead considers only individual words’ frequencies and co-occurrence patterns. It also naturally emphasizes “content-bearing” words (e.g., nouns, verbs, adjectives, etc.) in its representations while downplaying the influence of “low-information” words like disfluencies and discourse markers (in fact, these “stop words” are typically discarded altogether during preprocessing). This ends up making models like LDA actually quite well-suited to capturing conceptual similarities across modalities (e.g., between written content and transcripts of spoken content) since most of the linguistic features that tend to differ between modalities are ones that these models naturally disregard or de-emphasize.\nHowever, we want to emphasize that while we think these comparisons are interesting and relevant in the context of the reviewer’s particular question here, it is not our intention to make claims, provide commentary, or express preferences related to different model architectures in our manuscript. Importantly, we view these considerations around applying transformer models versus “bag-of-words” models to natural speech as a separate and more specific set of considerations than the general “sweet spot” notion we describe in our Discussion, since lecture transcripts are by no means the only form of content to which our framework could theoretically be applied (e.g., one could easily adapt our approach to estimate learners’ knowledge for concepts expressed in written media like books or scientific papers). By contrast, the idea that training a text embedding model on the specific content under consideration constitutes a “sweet spot” in terms of the “granularity” of its conceptual representations would hold true for any potential application of our framework.\nWe think using BERT as our example of a “generalist” model that “zooms out too far” is useful given its popularity as an off-the-shelf solution for generating text embeddings, particularly in behavioral research contexts. And seeing as multiple reviewers asked about the possibility of using LLMs in our framework, we suspect this example will be a useful one for readers as well. However, given the additional considerations we’ve touched on in the context of answering the reviewer’s specific question, we feel it merits further demonstrating here that this example is a reasonable one for the point we use it to illustrate in our Discussion—i.e., that the lack of “within-domain specificity” afforded by generalist models does indeed reflect the conceptual breadth of their training corpora, rather than something specific to “LDA versus BERT” or “bag-of-words models versus transformer models.” To this end, below we show an analogous set of plots generated using an LDA model we trained on a random sample of 100,000 English Wikipedia articles (a small but computationally tractable fraction of BERT’s training corpus). Of note, this generalist LDA model’s embeddings of our study’s course materials exhibit patterns similar to those we highlight as meaningful in BERT’s embeddings—namely, a reduced range of within- and across-content area correlations (with most being very high and all significant), diminished specificity in matching questions to periods of lecture content, and question embeddings that are less \"near to\" (i.e., often fall outside the convex hull of) the lecture trajectories. With sufficient time and computing resources to train an LDA model on a corpus even closer to the size and scope of BERT’s, we might expect these patterns to converge even more closely.\nWhat kind of test can be run to determine the correct level of granularity in the embedding space?\nThe idea of a specific “test” to determine the optimal level of granularity for the embedding space is an interesting one, and touches on a complex aspect of constructing text embedding models.\nAs described above, one key factor in determining the granularity of a model’s embedding space is the conceptual breadth of content included in its training corpus. For the particular role the text embedding model plays in our framework, the optimal breadth of training content is that which allows the model to devote its full representational capacity to capturing conceptual similarities and distinctions within the specific content for which we want to estimate learners’ knowledge, without “wasting” any of its capacity on representing other content outside the domain of interest. This is achieved by training the model solely on the specific course materials at hand.\nThe other key factor is the number of features (in the case of LDA, “topics”) the model is able to “use” to represent the content we want to embed (i.e., the dimensionality of the embedding space). Identifying an optimal value for this hyperparameter is comparatively less straightforward. Over the past few decades, there have been many attempts to define standardized, quantitative metrics for selecting an appropriate number of topics to characterize a given text corpus, with historically popular choices including held-out document perplexity, normalized pointwise mutual information, and various formalizations of “topic coherence.” We discussed and even implemented some of these measures in response to a comment from Reviewer 3 on our initial submission. However, there is growing consensus within this field that such automated evaluation metrics for topic models are fraught, and optimizing a model’s embedding space based on them tends to yield representations that are less useful in practice (e.g., Chang et al., 2009; Bhatia et al., 2017; Lipton, 2018; Doogan & Buntine, 2021; Hoyle et al., 2021; Stammbach et al., 2023). We tend to share the perspective described by Boyd-Graber et al. in their 2014 book chapter entitled “Care and feeding of topic models,” which emphasizes performing more holistic evaluations designed to assess the model’s usefulness for the specific task at hand. In our case, a “useful” model is one whose embeddings enable us to (A) distinguish between lectures and quiz questions about different content areas, and (B) match individual questions to relevant, temporally specific sections of lecture content, so that we can meaningfully infer learners’ knowledge of that content from their quiz responses.\nIn this way, the best “test” of whether a given model’s embedding space is appropriately “granular” for use in our framework is the sorts of diagnostic visualizations we show above and in Supplementary Figures 10 and 11. Indeed, this is the exact purpose we intended those figures to serve. In Supplementary Figure 10, we show how our LDA model’s embeddings of the lecture and question content differ from those obtained from a model whose embedding space is not sufficiently granular (i.e., BERT) in order to highlight how the latter model’s representations fail to meet the criteria for “usefulness” in our framework. In Supplementary Figure 11, we show how our LDA model’s embeddings differ from semantic representations that are too granular in order to highlight similar shortcomings. (While we didn’t explicitly fit a second model for Supp. Fig. 11, the Jaccard index of two text samples is equivalent to their similarity in an embedding space where every unique word is a separate binary feature.)\nIn our new Supplementary results section, we discuss the comparisons shown in these figures in greater detail, with a focus on what their specific visual elements indicate about a model’s usefulness in the context of our framework. We hope this can serve as a high-level guide that (along with the publicly available code we provide for fitting our models and generating these figures) can help researchers interested in adopting our approach apply and interpret these holistic “tests” themselves.\nAs with virtually all natural language processing tasks, our text embeddings benefit from some degree of manual oversight and hand-tuning. But fortunately, the speed with which an LDA model can be fit, evaluated, tweaked, and re-fit to tune performance generally makes this process fast and easy.\n3.In their explanation of their Fig. 6 results, the authors suggest that one possibility is that participants forgot some material: “If this forgetting happens in a relatively “random” way (with respect to spatial distance within the embedding space), then it could explain why some held-out questions about Four Fundamental Forces were answered incorrectly, even if questions at nearby coordinates (i.e., about similar content) were answered correctly.” (p. 18).\nThis seems to contradict their theory/finding regarding the “smoothness” of knowledge space. If forgetting happens randomly relative to the spatial location of knowledge, this evokes a kind of “Swiss cheese” knowledge space with holes randomly scattered about—in this case, knowledge would NOT be smooth, as the authors seem to claim. I can’t reconcile these two claims. Perhaps the smoothness visualized in Fig. 7 is a function of averaging/combining across participants, but then making the general smoothness claim would not be so warranted.\nWe appreciate the reviewer pointing out this inconsistency. We agree that the speculative explanation we had offered for this result (that within-lecture knowledge estimates do not predict success on Quiz 3 questions about Four Fundamental Forces) didn’t fit with our broader thesis about the “smoothness” of knowledge. While our interpretations of the results shown in Figure 6 have broadly changed (and, in our opinion, become more intuitive) now that we have addressed the issues described in our responses to comments 1a and 1b above, it’s worth noting that in every version of these analyses we have run to date (U-tests, conservatively biased likelihood-ratio tests, and now bias-corrected likelihood-ratio tests), these particular knowledge estimates have been the only ones we compute for Quiz 3 to not predict participants’ success on held-out questions. The consistency in this pattern of results across multiple changes to our analysis approach would seem to suggest that while estimated knowledge does in general predict success on Quiz 3 questions, there is some genuine effect in participants’ performance on the particular subset of questions used in this analysis that impedes our ability to distinguish between correctly and incorrectly answered held-out questions.\nWe did some deeper digging into the data for this analysis to try to determine what this effect might be. To summarize, we believe this consistent null result reflects a particular ceiling effect in participants’ quiz performance (rather than some broader aspect of their behaviors or knowledge). We describe this on pages 16–17 of our revised manuscript:\nSpeculatively, the Quiz 3 results suggest that the within-lecture knowledge estimates may be susceptible to ceiling effects in participants’ quiz performance. On Quiz 3, after viewing both lectures, no participant answered more than three Four Fundamental Forces-related questions incorrectly, and all but five participants (out of 50) answered two or fewer incorrectly. (This was the only subset of questions about either lecture, across all three quizzes, for which this was true.) Consequently, for 90% of participants, our within-lecture estimates of their knowledge for Four Fundamental Forces-related questions that they answered incorrectly leveraged information from at most a single other question they were not able to correctly answer. This likely hampered our ability to accurately characterize the specific (and by the time they took Quiz 3, relatively few) aspects of the lecture content these participants did not know about, and successfully distinguish them from the far more numerous aspects of the lecture content they now did know about.\nEssentially, as participants acquire more knowledge over the course of our experiment, a few things tend to happen that have opposing effects on our ability to predict their success on held-out questions. On one hand, with each successive quiz, the patterns in participants’ correct and incorrect responses will tend to become more faithful reflections of their “true” knowledge. Intuitively, this occurs because after participants view each lecture, their correct responses are by and large more likely to reflect “real” knowledge of the lecture content rather than spurious “noise” due to successful guessing based on vague intuitions. Additionally, as we describe on pages 18–19 of our revised manuscript, we suspect that after viewing the lectures, participants’ internal representations of their conceptual contents may become more closely aligned with the conceptual representations learned by our text embedding model, since the model was explicitly trained on those lectures’ transcripts. Together, if participants’ quiz responses come to better reflect their knowledge of the concepts being tested, and the relationships between the quiz questions’ embeddings come to better reflect how participants organize their internal representations (and/or knowledge) of those concepts, then our ability to predict success using estimated knowledge should increase with each quiz and be greatest on Quiz 3. Accordingly, one high-level pattern that has persisted across all versions of these analyses (irrespective of various biases that had nudged some results above or below the “p = 0.05” threshold in prior versions) is that the predictive strength of estimated knowledge tends to increase over successive quizzes (e.g., compare odds ratios & λ values between columns in each row of Fig. 6). This would also explain why our majority-significant results for Quiz 3 have been robust to these fairly substantial changes to our analysis approach.\nOn the other hand, as a participant gains more knowledge and answers more quiz questions correctly, our estimates of their knowledge become increasingly “saturated” (i.e., relatively high for all content) and the “signal” distinguishing content they do versus do not know about becomes somewhat weaker. This occurs because when a participant’s quiz responses contain fewer incorrectly answered questions, those responses provide more limited information about content that participant does not know about. The information contributed by these “negative samples” of a participant’s knowledge (i.e., questions they answered incorrectly) plays an important role in our knowledge estimates: while correct responses help identify underlying conceptual themes that are common among content a participant knows about, incorrect responses help determine which of those themes are specific to content they know about, by identifying conceptual themes that are also represented in content they do not know about. Within our framework, themes that are “common” among sets of quiz questions appear as similar patterns in those questions’ topic activations when we transform them using our topic model. Themes that are relatively “specific” to a particular set of questions then appear as patterns of topic activations that are more similar to each other than they are to those topics’ activations for other questions. When we estimate a participant’s knowledge for a given held-out question, that estimate gets “pulled” away from their unweighted remaining-proportion-correct score in the direction of their binary success on held-in questions that exhibit similar conceptual themes, with a “strength” proportionate to how specific those shared themes are (i.e., the correlation between the held-out question’s topic activations and a given held-in question’s topic activations, rescaled relative to that correlation value for other held-in questions).\nThat we have consistently observed significant positive relationships in four of the five analyses we perform for Quiz 3 questions indicates that, in general, our knowledge estimates can successfully distinguish conceptual themes that are specific to known versus unknown content even when relatively few “negative samples” are available because participants’ quiz performance is near ceiling. However, the nature of the “Within-lecture” analyses further compounds the challenges posed by this ceiling effect: since all questions we consider in these analyses pertain to content from the same lecture, they will tend to exhibit largely similar sets of conceptual themes overall (i.e., all questions’ topic activations will tend to be highly correlated), and the particular themes that tend to differ between correctly and incorrectly answered questions will present as relatively subtle differences between those questions’ overall patterns of topic activations. Our results suggest that for Birth of Stars-related questions on Quiz 3, we are still able to leverage these subtle differences to distinguish between known and unknown content. But for Four Fundamental Forces-related questions, for which our knowledge estimates could rarely leverage more than a single incorrectly answered question, we lacked sufficient information to reliably characterize these subtle, specific differences. We note that this is not a problem for our knowledge estimates per se, but rather it is a problem specific to validating our knowledge estimates. In other words, if a learner demonstrates perfect performance on every question, our framework will predict that they will also likely perform well on other questions (which seems to us a reasonable generalization). But since our testing framework requires us to show a contrast between correctly versus incorrectly answered questions, ceiling performers appear to be poorly predicted (since they have very little contrast between what they know most versus least)."}, {"round": 4, "reviewer_comments": {"reviewer_4": "The manuscript by Fitzpatrick and colleagues describes an application of text embedding models to measure and “map” conceptual knowledge during and after learning about some topics in physics (watching videos and then taking quizzes). I did not review prior versions of this manuscript but have been asked to assess the responses of the authors to prior rounds of feedback. Given this context, below I provide my own evaluation of the manuscript (mostly focused on the responses to Reviewer 3’s concerns), but I do not specifically suggest or request any further edits to the manuscript.\nFirst, it is clear that the authors have put a tremendous amount of work into responding to prior comments from reviewers. I think they have been about as responsive as possible and the paper is very well written. In many respects it is a beautiful paper. Reviewers 1 and 2 seem generally satisfied with the revisions. Reviewer 3, however, raised a number of concerns related to details of the statistical analyses, the specific models used, and the overall significance of the findings. My impression is that the authors have satisfactorily addressed Reviewer 3’s comments related to the statistical analyses and have appropriately contextualized the choice of model (with helpful comments about other models). However, I do share some of Reviewer 3’s concerns about the significance of the work—and these concerns are, I think, less addressable unless the scope of the paper was substantially expanded (which I think is infeasible). Thus, I have a mixed impression: the authors have done some very commendable work on this paper and there is certainly a wow factor to the methods and visualizations (which may be enough for the paper to have impact), but I do have concerns about the actual significance of the work.\nComments\n1. My overarching reaction to the manuscript is that it is more impressive than it is significant. Reviewer 3 shared a similar reaction: ““the authors’ algorithm is cool, but does it sufficiently advance us beyond the capabilities that already exist in large language models, which are available to students and instructors.” Reviewer 3 was perhaps more narrowly focused on the advance of the current methods relative to other large language models or to ChatGPT, but I think the point applies more generally. I might differ from Reviewers 1 and 2, but, to my mind, the empirical results are more of a sanity check than meaningful results on their own. That is, I don’t think these results speak to a theory or debate in the field of cognitive science. Rather, as the authors acknowledge, this is more of a methods paper and I think the empirical results mostly demonstrate that the method is doing what it “should do.” For example, showing that subjects know more about a topic after watching the video or that knowledge about a topic “predicts” performance for similar questions is something that most people would take at face value. Likewise, the fact that questions mapped to fairly specific time points in the lectures is something that almost needs to be true. If the questions are drawn from the lecture content (which they are), then it would be a red flag if the method couldn’t map questions to timepoints in the lecture. Thus, for each of these results, if the model failed to show these effects, it would be a reason to tweak the model, not a meaningful result. (And I think the authors would mostly agree with all of this). In fact, there are quite a lot of degrees of freedom that went into getting results that “make sense.” This is, of course, more evident in the rebuttal than in the actual manuscript.\n2. I think the question is whether the paper does enough, as a methods paper. (Again, I think this is where Reviewer 3’s concerns were, as well). As I note above, I think the authors have done enough on the statistics/analyses front to make a compelling case that the method “works.” That said, the paper does not introduce a new large language model, nor does it even use the latest models. Thus, I think the value as a methods paper comes down to the specific way in which existing models are leveraged to ‘map’ knowledge. Ultimately, I am not particularly convinced that there is much practical value in this method. The authors motivate the ideas in terms of teachers that might want to map what students know. Putting aside concerns about the actual feasibility of a teacher using this, the immediate question that came to my mind was: what is the advantage (gain) of this method relative to much simpler approaches. The authors compare their method against a strawman of a single test grade (e.g., 85%). The argument is that a single test grade makes it impossible to assess which specific topics or concepts a student has learned. That’s true, of course. But, for most of us that teach classes, it is very common (and almost a necessity) that we structure exams so that questions are organized around sub-topics. For example, we might deliberately include a certain number of questions per subject area. It is therefore quite trivial to have sub-scores by topic area. While I genuinely loved the visualizations that were presented in the current paper (again, it is beautiful work), I really struggled to see how or whether these visualizations provide more useful information than sub-scores on an exam. Even for the analyses that seek to predict performance on held-out questions, I was left wondering whether this method is any more sensitive than simple conditional probabilities based on sub-scores. For example, if a student answered the first 4 questions on a sub-topic correctly, what is the probability of answering question 5 correctly? What if they only answered 2 of the first 4 questions correctly? For a methods paper, I think that there is somewhat or a burden to demonstrate that the new method being presented outperforms existing methods. And, ultimately, I was not convinced that the current paper does this. Yes, the high resolution coordinate spaces are impressive and they certainly give the impression of being much more nuanced than a sub-score, but it is harder to see the actual application of that. Can higher-resolution feedback be provided to a student or teacher? And would this higher-resolution feedback make a meaningful difference to learning outcomes? In these respects, there is no clear evidence or argument that this new method is superior to much, much simpler alternatives.\n3. One way to demonstrate the value of the method would be to actually run a classroom experiment showing that students and/or teachers benefitted from the high-resolution maps compared to some simpler alternative. I realize the authors would argue that this is beyond the scope of the current paper (and I agree), but I think if the scope of the current paper is simply to demonstrate the method without demonstrating the actual value of the method, then it is necessarily a less significant contribution.\n4. Relatedly, Reviewer 3 raises the point that it’s not clear the current model would outperform ChatGPT in the core function of assessing conceptual knowledge. I tend to agree. I do get the authors’ points that their method can track and map learning in a way that ChatGPT isn’t capable of (I agree with this), but, to me, it really comes down to the potential value of the method. This gets back to my question of whether the high resolution maps can actually be translated to some meaningful feedback or intervention. But if the goal is simply to have a user-friendly, powerful method for assessing conceptual knowledge, I think ChatGPT is probably the more appealing option."}, "author_response": ""}]

Decision letter