Co-Creational Teaching of Natural Language Processing (2024)

Abstract: Traditional lectures have poorer outcomes compared to active learning methodologies, yet many natural language processing classes in higher education still follow this outdated methodology. In this paper, we present, cocreational teaching, a methodology that encourages partnership between staff and lecturers and show how this can be applied to teach natural language processing. As a fast-moving and dynamic area of study with high interest from students, natural language processing is an ideal subject for innovative teaching methodologies to improve student outcomes. We detail our experience with teaching natural language processing through partnership with students and provide detailed descriptions of methodologies that can be used by others in their teaching, including considerations of diverse student populations.

heading: 1 Introduction
text: Co-creational teaching is a methodology that involves an active collaboration between students and teachers in the learning process. Natural Language Processing (NLP) is a fast-moving area and it is my experience that students are increasingly demanding that the latest cutting-edge technologies are taught in the classroom. This presents a fundamental challenge to the teaching of NLP in that the teacher must deliver content that satisfies students’ demands while providing them with skills that will remain useful throughout their careers. This can also help to support students’ interest and the need to balance between cutting-edge neural NLP techniques and traditional statistical techniques. In this paper, we discuss the use of co-creational methods to enable students to explore the subject of natural language processing alongside the teacher, acting as subject matter expert, in a manner that co-creates the learning material.
Such a style of teaching provides several key benefits by encouraging students to engage in hands-
on examples of dealing with NLP challenges. It brings diverse perspectives into the classroom, representing the diversity of backgrounds and also reflecting the interdisciplinary nature of natural language processing. Co-creational teaching is a form of active learning (Felder and Brent, 2009) that has been shown to improve the retention of knowledge and deepen the comprehension in the subject area. This is achieved through activities where the teacher and student work together in group discussions, peer teaching and project-based learning activities. In a co-creational teaching environment, students provide feedback to each other and to the instructor throughout the learning process, which further leads to more effective learning outcomes. As NLP is a rapidly evolving field with new techniques emerging rapidly, this teaching methodology encourages experimentation and the sharing of findings with peers and teachers. In this way, such a teaching methodology builds a community within the classroom where everyone feels valued and encouraged to participate actively.
In this paper, we will discuss co-creational methods and how they can be applied to teaching natural language processing in higher education. These methods will act as an illustrative guide for other teachers applying these methodologies. We will also consider the challenge of evaluating students in such a co-creational environment, which has become a major issue in higher education due to the increasing adoption of generative AI technologies by students. We will demonstrate how these were applied in teaching a class of about fifty MSc students in a University of Galway course. Finally, we will reflect and discuss on the promise of cocreational teaching and how this can be applied across other settings in higher education.

heading: 2 Methods
text: Student participation in teaching has been described as a ladder (Martens et al., 2019) from
33
a fully dictated curriculum up to the level of students being fully in control as described in Figure 1. The ladder proposes a number of levels of student engagement:
Dictated Curriculum This is a traditional model of lecturing where the teacher dictates all content in the course
Tutor in control Students may be involved in a limited way, for example through quizzes or discussion groups, but the teacher retains full control over the content of the course
Limited Prescribed Choice In this case, the students can make some choices over the content of the course, however these are from a limited set of options provided by the teachers
Wide Prescribed Choice Similar to the above case, however in this model the teacher provides a wide range of options for the students to choose from.
Prescribed Student Choice Here the students are free to choose their own content, however, the teacher still has editorial control over the content.
Limited Free Student Choice Some areas of the content of a course are created without the
teacher’s control, while other areas are still at lower levels of this ladder
Partnership In this case, all areas of the course are created by discussion and negotiation between the teachers and the students
Student Control The teacher has no control over the curriculum and all areas are selected by the students
These levels describe a transition from a traditional lecture model of teaching towards full student control. As more control is given to the students, this will encourage more active participation and thus more engagement and better learning outcomes (Martens et al., 2019). However, as the teacher’s control is reduced it becomes harder to ensure that the curriculum is appropriate and topical and can be assessed effectively in the context of a university degree. We will now look at the methodologies that can be applied to teaching natural language processing and specific constraints or opportunities with this subject matter.

heading: 2.1 Co-design of Curriculum
text: Natural language processing is a rapidly changing area and the transition of knowledge from initial research proposals to teachable material in the
classroom is very short. Further, due to the wide coverage in the media and societal impacts of natural language processing research, I have observed that students are often highly knowledgeable about the subject area. As an example of this, a term like ‘language models’ has moved in the last few years from a term that was not familiar to many NLP researchers into a term that is now widely discussed in the media1. Due to the popularity and rapid speed of change of the subject area, natural language processing is an area that can benefit from the co-creational design of curricula.
There are a number of ways that students can be involved in the creation of a curriculum. Involving students from the initial stages is vital and this could be done by creating a pre-course survey that students could complete ahead of starting the course and could be administered through a learning management system. The first lecture within a course should be used to discuss the topics that could be chosen for the course and brainstorm and design the course syllabus. This provides students with opportunities to suggest topics, resources, projects, and assignments that align with their interests and career goals. Such a process can often be dominated by the most enthusiastic and motivated students in the class and thus in order to represent the class and to provide support for non-neurotypical students, asynchronous methods of feedback can also be implemented. One way of doing this is to create a poll that allows students to vote on the topics that will be covered as part of the course. This poll can be structured by the lecturer such that fundamental introductory material is covered earlier in the course.
Once, the curriculum of the course has been decided it is vital that this does not become fixed, as the students will learn more about natural language processing during the course and so their preferences will update with their learning. The teacher should create mechanisms for ongoing feedback throughout the course, such as anonymous surveys after each module or regular class discussions where students can voice their opinions and suggestions for improvement. Finally, this process should be repeated for every instance of the course in order to remain flexible and open to incorporating
1According to OpenAlex, 8.5% of papers that mentioned ‘natural language processing’ also mentioned ‘language models’ in 2003, by 2023 this had increased to 28.7%. Google n-gram reports a decline in the usage of the term ‘language model’ during the period 2005-2014
changes based on student feedback and evolving trends in the field of NLP. This can result in a curriculum that is dynamic and responsive to the needs of students and the rapidly changing landscape of NLP technology.
A key method for achieving this is backwards design (Wiggins and McTighe, 2005), where the students and teacher work together to identify desired learning outcomes and thus ensure that the educational goals align with the needs and interests of the learners. Throughout the course, it is important to continuously gather feedback from students by encouraging open communication and being flexible.

heading: 2.2 Peer Teaching
text: In order to achieve co-creational teaching, the teaching should move from a model where the lecturer leads all the content. In this way, the teaching methodology is similar to flipped classroom (Bergmann and Sams, 2012) approaches. Teaching through a flipped classroom generally leads students to score higher on both general and critical thinking exams (Talley and Scherer, 2013; Missildine et al., 2013; Mortensen and Nicholson, 2015). The standard way of delivering a flipped classroom teaching is through videos that the students watch in advance of the class, however, videos are not the only way to achieve this outcome (Uskoković, 2018). For a topic such as natural language processing, there is already a large amount of material available on sites such as YouTube and in many cases, this is of higher quality and more instructive than material that could be developed by a single lecturer. Thus, it is a great idea to incorporate this material in combination with material developed by the lecturer. Although, it is important to note that videos on YouTube often vary substantially in terms of the length and the assumed background knowledge of the viewers.
Further to enable a truly co-creational approach to teaching, the content to be taught should be developed in collaboration and ideally even by the students. In this way, the students achieve ownership over the material and gain a deeper understanding of the material. This jigsaw technique (Aronson et al., 1978) divides the content into smaller parts with each part assigned to a different group of students. In this way, each group becomes an expert on a part of the topic and can teach this to the overall group.
A particular challenge with this kind of teaching is that students can be unwilling to engage with this kind of teaching. As such, it is often the case that students are more keen to contribute in a textual form rather than in front of the class (Uskoković, 2018). Thus, the topic can be presented by each student developing a slide summarising the main ideas of the topic and in this way providing an opportunity for students to take ownership of their learning.

heading: 2.3 Group Work
text: Group work is one of the most important methods for enabling co-creational learning as groups by their nature involve students teaching each other and developing educational material. However, simply assigning students to groups and assigning a traditional project to them does not necessarily engender positive co-creational teaching outcomes. Instead, the teacher must develop a culture that encourages this form of co-creational teaching. One way to achieve this is through controversy theory which “posits that when students are confronted with opposing points of view, ... [it] results in more refined and thoughtful conclusions” (Johnson et al., 1998). In the case of cutting-edge fields such as natural language processing, there is plenty of room to discuss opposing approaches to tasks and the merits of different methods (e.g., LSTMs vs transformers). Other ways in which the group work can be structured for better outcomes include providing a joint reward for the group alongside an individual score and defining complementary roles within the group so that each student is clear about what they will achieve in the group. In this way, the problem of social loafing (Karau and Williams, 1993) (where one or more students in the group make little contribution) can be minimised as each student is responsible for a part of the project and thus the overall success of the project. Another idea can be to make sure that each student has to explain what they have worked on and learned to the other students and require this in the assessment of the group project.
While the students can be largely autonomous in the design and implementation of the group work, the role of the teacher is vital in setting the scope and in ensuring the effectiveness of such a project. For group work, Nokes-Malach et al. (2015) propose a theory of the Zone of Proximal Facilitation (ZPF), which hypothesises that collaborative suc-
cess depends on the relation between the task’s complexity and competence of the group and individuals. That is, if the task is so simple, that an individual in the group could do it by themselves, then group learning is unlikely to be effective. Conversely, if the task has too much cognitive load (Kirschner, 2002) for the whole group then the task will fail. As such, it is important that the teacher can orchestrate the task and intervene appropriately. This can be achieved by a specific orchestration tool, such as the one developed by Lawrence and Mercier (2019), which would allow the teacher to examine the progress of each group. However, for more open-ended co-creational projects a simpler method such as a weekly journal entry would be of value. Further, it is important that when the teacher intervenes in the work of the group this provides concrete help to the group and is not simply an interruption of the group’s work. As such, teachers must carefully review the orchestration and or journal to identify problems in the group work.
Working in groups is one of the most effective ways to promote active learning and it has been shown that cooperative learning promotes higher individual learning outcomes (Johnson et al., 1998). In particular, solving problems collaboratively increases engagement in STEM subjects (Freeman et al., 2014)2.

heading: 2.4 Disadvantages
text: While there are many advantages to this approach, there are some drawbacks to this approach. Firstly, these methods require a more spontaneous and dynamic approach to scheduling than would be required in a traditional course. This can clash with timetabling constraints at the institution and also make advanced planning more difficult, for example, with respect to exam questions. This may pose challenges for students, especially those from non-neurotypical backgrounds, who are more comfortable with planned lecture content. Secondly, the role of the lecturer is redefined in a way that changes the lecturer’s role. On the one hand, there is less need to prepare formal content, however, on the other hand, content must be prepared anew for each year that this course runs. As such, this kind of teaching may require more work from the
2As natural language processing is a subject requiring mathematics and programming, its teaching is more influenced by STEM, although we note that it is often delivered in linguistics departments at some universities
lecturer and certainly requires flexibility in terms of adapting to a new topic. Finally, while these methodologies should lead to more engagement from students, many students will not be engaged and responsive and as such there is a risk that these methods may produce poorer outcomes for less engaged students.

heading: 3 Assessment
text: One specific challenge with co-creational teaching is the assessment of the student work especially when the educational institute will require individual assessment of learners. When much of the work is done in groups, it can be difficult to assess the individual contributions. However, there are also opportunities for peer assessment in such environments to provide feedback beyond what would be possible by the teaching team of a lecture course.

heading: 3.1 Individual Assessment
text: Individual assessment of students is a particular challenge and the rise of generative AI has led to a crisis of academic integrity (Eke, 2023), that is still going on. The nature of teaching natural language processing requires that students interact with large language models and generative AI, as this is at the core of the subject. However, the temptation among students to use such methods for any continuous assessment is great and methods to reliably detect the use of such tools do not exist and may never exist (Dalalah and Dalalah, 2023). As such, many educational institutes are increasing the use of in-person exams to ensure academic integrity is maintained. However, these methods as well as being stressful for students, fail to produce objective assessments (Curzon, 2003) or provide feedback for students. Instead, co-creational methods would involve the students in the design of assessment materials and in the setting of goals that demonstrate learning. As such, continuous assessment should remain an important part of teaching and it is important that we find ways to continue this in the era of generative AI, including by allowing students to use these systems. For example, by structuring the assessment around the activities in the classroom, rather than essays or survey articles written separately, students must connect the assessment with their learning.
Some specific challenges that may be encountered are the requirements from professional bodies that certify courses, which may put certain restric-
tions on the way a course can be evaluated and these can make it hard to apply co-creational teaching methodologies. Further, the exam timetable of the institute may require that an exam is submitted before the curriculum has been fully developed in collaboration with the students, and this may also complicate the development of assessment material for the course.

heading: 3.2 Peer Assessment
text: An alternative to traditional, individual assessment that should be applied, as much as possible, to teaching is peer assessment. This model provides for a highly interactive method of assessment (Kollar and Fischer, 2010), that provides feedback and increases engagement with the material. It is important that the teacher scaffolds this correctly so that students gain valuable opportunities to learn from and support each other. Clear assessment that aligns with the learning objectives and expectations of the assignment or activity should be defined and, where possible, students should be given guidance on how to give constructive feedback to minimize bias and promote fairness. These guidelines should emphasize the importance of providing both strengths and areas for improvement, and encourage students to offer suggestions for how their peers can enhance their work. This can even be incorporated into the grading process such that students receive marks not only based on the feedback from their peers but also on the feedback they provide. The teachers’ role is also important and they must follow up with students after the peer assessment process to discuss the feedback they received, address any questions or concerns they may have, and provide additional support or guidance as needed.

heading: 4 Experience
text: We applied these methods of co-creational teaching to a lecture course on ‘Advanced Topics in Natural Language Processing’ (CT5121) taught at the University of Galway in the second semester of the academic year 2023/24. Students had already taken a one-semester introductory course on natural language processing and so had a broad familiarity with the area and this helped in terms of choosing topics for the course. The course was taken by 47 students in MSc programmes on Artificial Intelligence, Data Analytics and Cybersecurity over the course of 12 weeks. These students were mainly
graduates of computer science and other STEM programmes and were predominantly international students. The course was delivered through flipped classroom lectures, where the lecturer and the students worked together to define topics, and through open-ended group projects.
The syllabus was defined through discussions with the class. The lecturer suggested an initial list of topics that were then discussed in the class and updated based on student feedback. The lecturer organised these into three classes of theories (e.g., ‘Recurrent Neural Networks’), methodologies (e.g., ‘Named Entity Recognition’) and applications (e.g., ‘Question Answering’) and the polls initially contained only theoretical topics and the later polls introduced more methodology and application topics and removed theoretical topics (based on receiving fewer votes in the earlier polls). The topics for each week were presented to the class and discussed and this in a few instances led to updates in the topics from the selection chosen by the lecturer. No specific material was provided to help choose the topics, but the lecturer guided the class discussion of the topics to encourage new suggestions. The class suggested a number of topics and these were then put to the class through an open vote on the learning management system Blackboard. The final resulting syllabus is shown in Figure 2, and this process was repeated for the first five weeks of the semester3, and the final five topics were fixed by
3Due to public holidays in Ireland, two lectures were can-
a single poll. This was due to constraints on the assessment of the material by means of a written exam. Each lecture was prepared by the teacher finding appropriate video material on YouTube, as well as writing some outline notes on GitHub 4. The students were instructed to review these materials before the lecture and the classes were then structured around open discussion and interactive exercises on the topics. For example, in one lecture the students were divided into four groups and competed to implement various few-shot and zero-shot methodologies on a single dataset.
The other main component of the course was an open project, that the students completed in groups of their own choosing, with groups ranging in size from one to five participants. The students discussed these projects with the teaching team and this feedback was taken by the teaching team and used to adjust the content continuously throughout the semester. An end-of-module survey was deployed through the learning management system, however, participation was poor and this did not provide any useful information on the success of the teaching methodology. The students were encouraged to find their own topics and to structure their own learning. The project work was assessed by two written essays and a final presentation. The two written assignments were approximately 1,000- 1,200 words and they were marked by the primary lecturer, with feedback given to the students. A flexible policy was applied to the submission deadline for these assignments and the majority of the class submitted these assignments later than would be planned in order to provide constructive feedback. The presentation was made to the lecturer and two teaching assistants and consisted of a 10- minute presentation or demo, as the groups saw fit, the mark was agreed between the lecturer and teaching assistants. Groups were encouraged to discuss among themselves and with other groups and provide feedback. Finally, the course had a final written exam, due to institutional requirements.

heading: 4.1 Lessons learned
text: Overall, the course received strong positive feedback in terms of student engagement and the overall outcomes of the students were strong in the written exam, showing that they had benefited from choosing their own teaching and the active learn-
celled during the semester 4https://github.com/jmccrae/2024-CT5121
ing provided through the course. Still, there were some issues, especially related to the engagement of students with the material that will be improved in future iterations of the course. Firstly, from the lecture sessions, it was clear that the video material was not generally being watched by students ahead of the lectures. This may in part be due to the content being available on another platform than our learning management system and the variability in the length of the material selected. Further, feedback throughout the course was not taken advantage of by many students, who only engaged with the project near deadlines. This was particularly notable in the peer group assessment which was very infrequently used. In future instances of this course, a weekly journal and formal marks being assigned by peer assessment will encourage more engagement. Finally, several students chose to work in groups of one (e.g., alone), and these students were much less engaged with the teaching, so in future groups of two will be the minimum group size. In summary, this experience focused on the techniques of co-designing the curriculum and the use of group work and did not apply some of the peer teaching techniques discussed above, but this will be a goal for future instances of this course. Student choice, in this case, was limited but free in the topics, however, students in most cases accepted the lecturer’s suggestions regarding topics leading to a curriculum that was more prescribed, yet still quite different than the curriculum that would have been chosen by the lecturer. The next instance of the course will focus on improving peer instruction and applying some of the techniques discussed in this paper.

heading: 5 Discussion
text: It has been widely concluded that the traditional lecture is the least effective way to communicate with learners (Laurillard, 2013; Bligh, 1985). For this reason, there has been an increasing focus on active learning approaches which can processes and outcomes in higher education (Kuh, 2008). Further, it has been noted that students learn best when they become their own teachers (Hattie, 2008) and this is one of the key objectives of co-creational teaching and learning.
A number of methods have been proposed to encourage student participation in teaching. Firstly, design-based research is a methodology that involves iterative cycles of design and implementa-
tion emphasizing co-design and co-implementation with stakeholders. This represents the lower rungs of the co-creation ladder as depicted in Figure 1. Participatory design (Scheer et al., 2012) involves students to a higher degree, where they play a central role in defining the curriculum. Co-creation involves educators and learners working as equal partners in the education process to create learning experiences that meet the needs of all participants. All three terms have been used for active collaborative learning practices but differ in the focus and degree of participation of the students in their learning. In fact, methods that focus on participation are not novel and can be seen in the dialogic methods of Aristotle or the progressive education movement of the late 19th and early 20th century (Dewey, 1916)
One of the key goals of co-creational learning is to generate critical thinkers (Freire, 2000) who take responsibility for their own learning (hooks, 2014). As such co-creational teaching empowers students to collaborate with their teachers. However, cocreation is an open-ended model requiring teachers to give up “complete creative control” (Uskoković, 2018). This can be challenging and can be seen as almost ‘counter-cultural’ (Cook-Sather et al., 2014) in modern higher education environments. The goal of teaching in this manner is to promote equality and partnership between lecturers and students. In this way, it is important to see the teachers as learners as well, and in fact, the syllabus selected by the students in the course described above went beyond the lecturer’s (and lead author of this article’s) expertise in several areas of natural language processing. As such co-creational teaching reframes education where very often the students are seen as problems to solved (Sambell et al., 2012), into a space where equality is natural. However, it is important to understand that student participation does not replace teachers’ expertise (Breen and Littlejohn, 2000), and there is naturally an imbalance of knowledge between the lecturer, who is often a subject matter expert, and the students who have limited knowledge of the subject area. As such, the teacher should retain executive control and work with the students to direct them into areas that are interesting and valuable to study. Finally, it is important to note that co-creational methods can often threaten students as well (Bovill et al., 2011), as it breaks from the usual passive consumption of material that they have experienced in their studies
so far. As such, this can lead to poorer engagement for some students who see active and collaborative methods as merely extra effort. It is important to make the benefits of such teaching methods clear to students and not to compel any students to take part in these methods. Further, many aspects of co-creational teaching can be conducted in liminal spaces that do not affect the main delivery of teaching but support the teaching. The results of this can lead to student testimonies such as “I am finding myself being more understanding of my professor’s struggles” (Bovill et al., 2011), illustrating the value of this approach.
In the particular context of natural language processing, the role of co-creation is important as this is a subject for which there is wide interest and thus a wide amount of educational information available on the web. As such, to connect with the ‘YouTube Google-eyed generation’ (Ashraf, 2009), it is important to situate the teaching within this context and thus to help students find their ways to valuable material. We find that students will consult YouTube anyway and this can lead to conflicts with attempts to impose a top-down curriculum. Further, providing students with suitable recorded material as well as summary notes can help substantially with their learning of the topic.
Finally, students with Autistic Spectrum Disorders (ASD) and other neurodivergent traits are particularly attracted to STEM subjects (Wei et al., 2013) and can be expected to be seen in higher proportions in classrooms teaching natural language processing. Many of the characteristics of people with ASD can be in conflict with the collaborative and participatory methods proposed in this paper. In particular, many students with ASD have issues with personal interaction and do not like learning through videos and social situations. As such, it is also important to allow these students to interact through text reports or prepared material where possible. Further, clear guidance and instruction (Stuurman et al., 2019) are vital to ensure that all members of a class understand the task as some students do not learn well by examples. In group work, which is a key method of co-creational teaching, students with ASD can perform well by focusing on specific tasks (Wareham and Sonne, 2008), especially those that focus on details such as testing. Conversely, the co-creational method can act as a key method for involving students with ASD in the classroom, by allowing them input over
the structure of the programme, including not just only the curriculum but also the teaching methods, and by assisting in the development of collaboration guidelines that can outline how all students interact in the course of their study of natural language processing.

heading: 6 Conclusion
text: This paper has presented the co-creational methodology for teaching and focused on how it can be applied to the teaching of natural language processing in a higher education setting. We defined three key pillars of teaching: curriculum co-design, peer teaching and group work and showed how partnership with students can be achieved through these methods. We also considered how student performance can be evaluated using such approaches, especially in the current academic integrity crisis. This approach was tested in an MSc course and the results showed good engagement, with several key areas that can be improved. We also considered how these approaches can be adapted to students in particular those with neurodiversity.
Limitations
This work focuses on a teaching methodology and is primarily a theoretical work. This work has only been evaluated in a single setting and further application of this methodology and quantitative analysis would support this work.
Ethics Statement
The anonymity of all students has been preserved in this report. We do not see any other ethical issues with this work

heading: Acknowledgements
text: This work has been supported by Science Foundation Ireland under Grant Number SFI/12/RC/2289_P2 Insight_2, Insight SFI Centre for Data Analytics and the ADAPT Centre for Digital Content Technology (grant number 13/RC/2106 P2).

heading: Future Work
text: Still, there were some issues, especially related to the engagement of students with the material that will be improved in future iterations of the course. Firstly, from the lecture sessions, it was clear that the video material was not generally being watched by students ahead of the lectures. This may in part be due to the content being available on another platform than our learning management system and the variability in the length of the material selected. Further, feedback throughout the course was not taken advantage of by many students, who only engaged with the project near deadlines. This was particularly notable in the peer group assessment which was very infrequently used. In future instances of this course, a weekly journal and formal marks being assigned by peer assessment will encourage more engagement. Finally, several students chose to work in groups of one (e. g. , alone), and these students were much less engaged with the teaching, so in future groups of two will be the minimum group size. In summary, this experience focused on the techniques of co-designing the curriculum and the use of group work and did not apply some of the peer teaching techniques discussed above, but this will be a goal for future instances of this course. Student choice, in this case, was limited but free in the topics, however, students in most cases accepted the lecturer’s suggestions regarding topics leading to a curriculum that was more prescribed, yet still quite different than the curriculum that would have been chosen by the lecturer. The next instance of the course will focus on improving peer instruction and applying some of the techniques discussed in this paper