Logan Rath

State University of New York, Brockport, NY, United States

Peter Kalenda

State University of New York, Geneseo, NY, United States

# DOCUMENT ANALYSIS AND INFORMATION BEHAVIOR OF PRE-SERVICE TEACHER PERCEPTIONS OF CHATGPT-GENERATED LESSON PLANS

## Abstract

This study examined documents produced by generative artificial intelligence as well as users' perceptions of the usefulness of the documents themselves. The researchers performed document analysis on two generations of lesson plans created with ChatGPT versions 3.5 and 4.0. Additionally, three semesters of pre-service childhood educators also reviewed the lesson plans for accuracy and adherence to their course assignment goals. This poster will share findings and trends as well as implications for information practice use of as generative artificial intelligence increases in higher education.

## Introduction & Background

Generative Artificial Intelligence (GAI) has become a popular tool for in-service teachers since its introduction in 2022 (Mittal et al., 2024). Like any technology, though, adoption was not instantaneous (Burke et al., 2018). The researchers of the present study noticed a range of reactions, from fear to excitement when discussing implementation. Upon their own exploration of the use of ChatGPT to complete a lesson plan in elementary science methods, the researchers were quite surprised that an initial review of the output of a prompt resulted in a lesson plan that would have received a passing mark if it were to be submitted by a student. One researcher, with an information science background, came to the conclusion that the output of several similar prompts needed to be analyzed so that one can determine the extent to which GAI serves as a sociotechnical tool for information retrieval and creation of documentation. Based on the results of this line of inquiry, an inventory of information literacy practices can be created which may need to include new and not-yet defined practices for navigation of this new information landscape.

The output of ChatGPT qualifies as a document in the ordinary sense (Buckland, 1997). However, it is unclear the extent to which the document is considered socially useful in the context of an elementary science classroom. That is, can the generated lesson plan conform to the requirements of the assignment as written. This study aims to answer this specific question. The specific research question(s) for this study are:

1. To what extent, if any, do the documents output by ChatGPT align with the assignment requirements without iterative prompt engineering?
2. What are the perceptions of pre-service teachers when evaluating these documents?

3. What, if any, are the differences between documents generated using version 3.5 and 4.0 of ChatGPT?

## Methods

In addition to 42 documents, this study involves human participants to answer the research question related to pre-service teacher perceptions. Participants in this study were students in a pre-service combined science and mathematics education methods course over three semesters at a public higher education institution in the northeastern United States. Participants followed a pre-posttest design to measure their perceptions of GAI to develop a lesson plan that would meet the requirements of the assignment. Between the pre- and posttests, the students underwent an educational intervention to evaluate the curriculum produced. All students in all sections of both semesters participated in this process. However, only students who actively consented to their data being used were included in the study.

The specific documents evaluated by pre-service teacher candidates were lesson plans reviewed that were researcher generated. Each semester had a set of 21 unique documents consisting of a combined set of standards that were used to develop a standardized prompt for submission to the GAI model. Each prompt was submitted to the versions of ChatGPT available during the semester where the analysis took place. During the course of the study two versions of ChatGPT were published (version 3.5 and version 4.0). The results in 42 total documents. These documents were analyzed for completeness by the participants for adherence to the social conventions of inquiry-based science and math instruction (Kalenda et al., 2024).

## Data Analysis

For the pre-posttest design, a two-tailed t-test was performed to determine the extent to which perceptions of the lesson plans changed for each criterion of the assignment. Content analysis (Julien, 2008) was also used by the researchers to examine the qualities of each lesson plan. Comparisons between documents were performed to demonstrate similarities and differences across documents at the same grade level as well as between different versions of the model. Descriptive statistics were used to report alignment where possible.

## Conclusion

The poster will focus on any significant findings as well as outcomes of the document analysis of the various lesson plans. By documenting the qualities of the specific lesson plans generated through the use of GAI, information science researchers can begin to debate the extent to which these documents fit with more current definitions of document with emphasis on the social as well as how these documents can be useful to inform the information seeking behavior of pre-service teachers.

## References
Kalenda, P. J., Rath, L., Abugasea Heidt, M., & Wright, A. (2024). Pre-service Teacher Perceptions of ChatGPT for Lesson Plan Generation. Journal of Educational Technology Systems, 53(3), 219-241. https://doi.org/10.1177/00472395241301388

Buckland, M. K. (1997). What is a “document”? *Journal of the American Society for Information Science*, 48(9), 804–809.

Burke, P. F., Schuck, S., Aubusson, P., Kearney, M., & Frischknecht, B. (2018). Exploring teacher pedagogy, stages of concern and accessibility as determinants of technology adoption. *Technology, Pedagogy and Education*, 27(2), 149–163.
https://doi.org/10.1080/1475939X.2017.1387602

Julien, H. (2008). Content analysis. In L. M. Given (Ed.), *The SAGE encyclopedia of qualitative research methods* (2nd. ed., pp. 120–122). SAGE.

Mittal, U., Sai, S., Chamola, V., & Sangwan, D. (2024). A Comprehensive Review on Generative AI for Education. *IEEE Access, 12*, 142733–142759. IEEE Access.
https://doi.org/10.1109/ACCESS.2024.3468368