CONTEXT The research study was conducted at a contact -based, research- intensive university in South Africa, where the faculty of engineering has adopted a feedback -feedforward approach to improving engineering pedagogy through theoretically -supported, interdiscipl inary and community - of-practice approaches. The outcomes -based curricula are designed to explicitly align teaching/learning activities, the intended learning outcomes and assessment tasks. The Covid -19 emergency remote teaching (ERT) phase has raised the question of the disjuncture between student perceptions and assessment performance during independent, remote learning. PURPOSE OR GOAL A faculty -wide research initiative to determine how undergraduate engineering students were experiencing ERT revealed significant systemic challenges and heightened academic stress. Of particular concern in 2021 is the 2nd year cohort, whose entire 1st year was under ERT conditions. Poor first term assessment performance suggested the need to investigate not only how student s were studying, but their perceptions of their practices and efforts in relation to their perceptions of course requirements, and consequently their performance. APPROACH OR METHODOLOGY/METHODS A mixed -method survey -based approach was used to assess second year students’ perceptions of a design- based module. The surveys were sent out when it became clear that performance was going to be substantially poorer than expected for their first in- person and closed- book assessment after ERT. The samples were taken after the assessment, after the model answers lecture, after the marks were published, and again after an intervention. The 2020 marks were compared with the last in- person assessments from 2019. Out of the 280 students, 142 responded to the survey. ACTU AL OR ANTICIPATED OUTCOMES Students overestimated their marks after writing, even after seeing the model answers. Two thirds reported the paper as difficult, which reduced to 58% after the model answers, and 74% after releasing the marks. Two thirds said online lectures prepared them sufficiently, but after the marks only 45% did. After a reflection- in-action intervention, 81% considered them sufficient and the error in estimated marks for the next assessment reduced by 41%. Despite 97% engagement with the lectures and 96% claiming to have done the tutorials and practicals on their own, only 38% used the Q&A forums, and not a single student made an appointment with the lecturer. CONCLUSIONS/RECOMMENDATIONS/SUMMARY While constructive alignment is a common pedagogical approach, it does not explicitly include alignment to student abilities or perceptions. In contact -based, socio- culturally mediated contexts, educators may tacitly be responsive to (mis)conceptions to enhance alignment between student abilities , expectations and intended course outcomes. We suggest, in this paper, that a constructive alignment model needs to include methods to overcome self -efficacy gaps, given that we need to produce critically -thinking, confident, and capable graduates. KEYWOR DS Evaluative judgement; Constructive alignment; Engineering education; Emergency remote teaching. Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Booysen and Wolff, 2021 Introduction The International Engineering Alliance Graduate Attributes (2013) for a professional engineering graduate (Washington Accord) stipulate outc omes designed to enable the holistic development of graduates who are capable of solving ‘complex problems’. These outcomes extend across the knowledge, skills and ‘dispositions’ continuum, and are intended to ensure that graduates engage in engineering ac tivity which is “carried out responsibly and ethically” in “meeting the needs of people, economic development and the provision of services to society” (IEA, 2013, p. 1). The changing nature of the engineering profession has seen two decades of global engineering curriculum reform, with greater attention to the explicit integration of appropriate ‘knowledge, skills and attributes’ (DHET, 2013). Competencies in these areas are specified in engineering standards, which are supported by the global engineering accords and the IEA Competency Profiles (2013). Increasingly, pedagogical strategies such as project- and/or problem -based learning are intended to enable the holistic development of engineering knowledge, skills and attributes in the context of real -world problems. Recent research on how 21st century engineering knowledge, skills and attributes are holistically developed demonstrates the importance of explicitly teachin g the different ways of thinking and doing in relation to the different engineering disciplines (Wolff and Booysen, 2019). Students need to be taught how to recognise conceptual and contextual ‘codes’ which require simple to complex approaches (Pott and Wolff, 2019). Learning to code shift using appropriate strategies could be termed ‘critical thinking’ (Douglas, 2012; McPeck, 2016), which is based on the interpretation, analysis and evaluation of a problem situation. Critical thinking is the basis of engineering judgement, an under -researched competency (Tai et al ., 2018). Our Higher Education (HE) system is replete with examples of the continuous exercise of ‘judgement’: from the selection of materials judged to be necessary in the curriculum to the eventual judgement of student performance by way of assessment. In other words, the educational space offers an ideal space for the modelling of judgement practices as well as their development. Research on undergraduate student perceptions of their performance, however, reveals a significant disjuncture between student and educator judgements. By way of example, two studies on students’ perceptions of their computer skills versus their actual abilities reveal a notable discrepancy (Grant, 2009; Peng, 2009), with students consistently believing they are more able than they actually demonstrate in practice. This misconception is further evident in the “disconnect between students and faculty in expectation, perception, and reasoning behind academic evaluation” (Tippin et al ., 2012). Another common misconception (or perception) is that the exam paper differs significantly from exam practice papers (Young et al., 2019), and students often demonstrate “a low degree of success in predicting their success on a given problem” (Gulacar and Bowen, 2014). Confidence is a key element in effective judgement, impacting on decision-making (Christopher and Herbert, 2021). Bandura (1977) links confidence in effectively executing action (or taking decisions) to what he terms ‘self-e fficacy’. Additional judgement-related factors which impact on decision -making are fear, avoidant behaviour, and motivation. Motivation is linked to ‘ goal-setting’ (Schunk and Gunn, 1985), a key cognitive process which can be modelled through pedagogical s trategies and enhance self-efficacy. While acknowledging the complexity around what we are terming ‘evaluative judgement’, the ability to confidently and independently interpret practices and criteria (in order to make effective decisions ) is not only a central engineering graduate attribute, but one that is necessary throughout life (Boud and Soler , 2016). But given that ‘evaluative judgement’ is so poorly researched and there is little empirical work on how to develop students’ evaluative judgement (Tai et al., 2018), the question for this paper is what do we know about engineering student judgement in a particular context and how can we foster its development? The starting point for many educators in designing curriculum and pedagogy to facilitate the development of equipped, problem -solving critical thinkers, is the Constructive Alignment (CA) Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Booysen and Wolff, 2021 framework. CA is the alignment between a constructivist understanding of learning and the design for teaching (Biggs, 1996). The CA framework is intended to enable educators to explicitly link intended learning outcomes to the associated learning activities and the eventual assessment instruments. If our intention as educators is to design and engage in learning that enables the development of holistic engineering graduate outcomes, and our design instrument is that of CA, we argue that ‘evaluative judgement’ is a hidden outcome (Fitzpatrick, 2009), dependent on student and staff perceptions and expectations being made more explicit. The experience of emergency remote teaching (ERT) (Hodges et al., 2020) at a residential university in South Africa highlighted a significant disjuncture between student perceptions of their performance and that of lecturer expectations. Drawing on Vygotsky’s sociocultural theory of l earning (Kozulin, 2002), the paper uses a particular case study in an electrical engineering department to examine student perceptions of their performance during ERT and an intervention strategy to improve evaluative judgment going forward. Drawing on thi s study, we illuminate student performance perception patterns and suggest an more contextually nuanced review of the Constructive Alignment framework that better enables potential graduates to develop ‘evaluative judgement’. Theory Learning happens in soc io-culturally mediated settings (Kozulin, 2002), where ‘social’ refers to the relevant stakeholders (such as, but not limited to, students and teachers) in a particular ‘community of practice’ (Lave and Wenger, 1991). This community, then, refers to the ‘cultural’ aspect, where the culture of the community is that which constitutes the ‘rules of the game’ in a specific field. Sociocultural learning (to survive in the world) happens from the beginning of life, where human beings learn the rules of the game through modelling and repetition. Formal learning is facilitated when human beings engage in activities (whether perceptibly active or passive) that draw on mediating tools and resources, such as texts, knowledgeable others, artefacts and events. An iterative, scaffolded learning process can enable ‘cumulative learning’ (Maton, 2013), the connection of concepts to contexts through forms of application. Kolb (1984) describes this cycle as concrete experience, reflective observation, abstract conceptualisati on, and active experimentation. The concept of reflection in this learning cycle is taken further by Schon (1983), who usefully differentiates between reflection- in-action (during) and reflection- on-action (after). It is this concept of reflection that is vital to the development of ‘evaluative judgement’ (Tai et al., 2018) . In order to enable learning to happen, educators use artefacts to design their curriculum and pedagogical strategies. At each of these stages, a process of recontextualisation (Bernstei n, 2000) takes place, where what is selected and how it is to be taught is dependent on a range of stakeholders and contextual factors. A common instrument in aligning the what and how is Bigg’s (1996) Constructive Alignment (CA) model in which the focus i s on the relationship between objectives, ‘appropriate’ teaching activities and assessment. The question of ‘what is appropriate’ is highly contextual and suggests it is necessary to consider Vygotsky’s ‘Zone of Proximal Development’ (Kozulin, 2002). Although used to describe the distance between actual and potential development in childhood learning, Vygotsky’s aim in formulating the ZPD was the development of theoretically -based pedagogical interventions, responsive to the individual needs of learners (Shabani et al., 2010). Global massification in the HE system, and particularly in resource-constrained contexts (such as this study) , makes individually -focused tuition unlikely. The alignment of objectives to outcomes, teaching activities and assessment, thus, does not explicitly reflect the learner position in a particular context and his/her ZPD. One way to address the lack of attention to the individual student’s position, perspective or needs in large class contexts is to draw on students themselves as ‘ mediation resources’. Furthermore, the educational activities in and of themselves represent mediating ‘artefacts’, potentially enabling Kolb’s experiential cycle (1984). A key mediating artefact is the Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Booysen and Wolff, 2021 assessment event. Given the examples from the literature on observations of disjunctures between student perception and lecturer expectations, which manifest in assessment performance, assessments present a significant opportunity to 1) determine individual students’ ZPD, 2) enable reflective, experiential practice and 3) teach students ‘what counts’. Engaging with student perceptions of and reflections on assessment processes can enable educators not only to enrich their own pedagogical and curriculum desi gn, but also construct a more holistically aligned framework. Drawing on socio-culturally mediated learning concepts, this paper proposes that we cannot get the student to achieve the ‘outcome’ if we do not know where the student is at. Secondly, the student cannot truly develop if he/she also doesn’t know where he/she is at. If a key objective in engineering education is to produce critical thinking problem -solvers, how do we explicitly enable the development of evaluative judgement? Context The research study was conducted at a residential , research-intensive university in South Africa, where the faculty of engineering has adopted a feedback -feedforward approach to improving engineering pedagogy through theoretically -supported, interdisciplin ary and community -of-practice approaches. The outcomes -based curricula are designed to explicitly align teaching/learning activities, the intended learning outcomes and assessment tasks. In March of 2020, with the looming onset of the Covid-19 pandemic, South Africa entered one of the most severe lockdowns for any country, due to fears of an overburdened health system in a developing country. The lockdown commenced in the middle of the first semester, with all teaching moving online in an emergency remote teaching (ERT) (Hodges et al., 2020) phase. Although the transition was managed as well as reasonably possible, this modality inevitably severed important feedback loops between students, teaching staff and content. Moreover, the lockdown, which enforced teaching & learning-distancing, commenced before the first- semester exams started, and also encapsulated the second semester exams. This had the unfortunate consequence that the first-year cohort from 2020 wrote online-only exams. In 2021 ERT was augmented w ith limited in -person practicals and tutorials, and examinations . The 2020 ERT phase had raised the question of the disjuncture between student perceptions and actual assessment performance during independent, remote learning, which appeared particularly prevalent among the first -year cohort of 2020. This study focusse s on a second-year Computer Systems (CS) course presented by Electrical and Electronic Engineering (E&E). The course is compulsory for a nominal 280 students every year, and builds on the w ork covered in a first-year Computer Programming course, in which many students encounter software development for the first time. CS is also the first course in which the students are expected to transcend the boundaries of theoretical knowledge and design logic circuits. The main focus of the CS course is to teach students about binary and hexadecimal number systems, digital circuits, Boolean logic, combinational circuits, sequential logic, state machines, and assembly language. The course is heavily scaffolded by practicals which contribute to a continuous assessment grade. However, the lion’s share of the students’ marks are made up by a written Assessment 1 (A1) and written Assessment 2 (A2), contributing 30% and 50% respectively. The focus of this paper is the two -hour, written A1 exam, the first in-person assessment the students wrote after a year of pandemic -inflicted online assessments. Methods A mixed -method, online, survey -based approach was used to assess second year students’ perceptions of the design-based module. The survey consisted of seven Likert-scale questions to assess the students’ perception of the assessment, their personal preparedness, and resource use. Four separate surveys were taken to assess students’ initial and subsequent Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Booysen and Wolff, 2021 perceptions. The first survey was taken within two days after the assessment. Students were asked to guess their mark, to state whether they believed the CS online teaching prepared them sufficiently for the assessment, and whether they thought the paper was di fficult. The second survey was taken after an online video was posted in which the lecturer worked through the A1 model answers (called a “memo”) in detail using a tablet and electronic pencil and recording the screen. The same three questions were posed i n the second survey. The mark estimates from these two surveys were compared with the results from the 2019 CS class (predating any Covid-19 impacts) and also with the achieved A1 marks for the research study cohort. A third survey was conducted after the marks were released, and students were asked again if they thought online teaching prepared them sufficiently for the assessment and also whether they thought the paper was difficult. After the A1 model answers were shared and the results released, an intervention was designed to engage with the course content “in full view of” the students using an online ‘flipped classroom’ approach. Students were encouraged to share their own screen with their own problems from the practical assignments. In full view of all the other participants, the lecturer would then guide the courageous student in real -time as if they were in an in-person practical, or answer questions they may have. A fourth and final survey was conducted after the final major individual assessm ent (A2) was written six weeks later. The 2020 marks were compared with the last in-person assessments from 2019, which were unaffected by online learning. Out of the 280 students, 142 responded to the survey. All data were anonymised and collaboratively analysed using Microsoft Excel. Discussion of Findings Student performance perception Figure 1 shows the responses to the four surveys on the level to which the online lectures prepared the students and the difficulty of the paper. The initial responses jus t after writing A1 indicated that 68% of the students believed the online lectures prepared them sufficiently, which matches closely with the 69% of them stating the same after watching the memo lecture. However, after receiving their below -par marks, this number reduced to a mere 45% of students believing that the lectures prepared them sufficiently. This shift is interesting since it happened after the students watched the memo lecture. This supports the common disjuncture between student self-efficacy perception and the evaluative judgement of the educator. What is of further interest is the shift back to a high 81% after the students wrote A2, which included the course content for A1. However, A2 was preceded by the Q&A lectures with high levels of probl em-solving engagement. Figure 1 also shows the perception of A1’s difficulty. After writing, two thirds believed the paper was difficult. This number reduced to 58% after they had the opportunity to observe the lecturer working through the question paper. A similar (but inverted) trend is seen after the marks were released, with 74% of students claiming A1 was difficult. Again, this finding is supported in the literature on student perceptions on the difference between actual exams and their expectations ( Young et al ., 2019). Figure 2 shows the comparison of marks students thought they would receive and what they actually received at the different assessment stages. As stated, the expected performance was poor - the 2019 median mark for A1 was 56%, which is in line with expectations from previous years. However, the median actual mark for 2021 was only 33% and mean 34.5%. The respective 75th and 25th percentiles were 43% and 24%, compared to 64% and 44% in 2019. It is worth noting that several independent moderators -- both as part of moderation before A1 was written and after the poor performance was reported to the home department as part of a post-mortem investigation -- confirmed that there was no significant difference between the Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Booysen and Wolff, 2021 course content and types of assessment questions in the previous years and 2021 examinations. Immediately after A1 was written, the median guessed mark was 50% and the mean 48.8%. The 75th and 25th percentile guesses were 60% and 40% respectively. After watching the lecture detailing the memorandum, but before the marks were released, the median guessed mark decreased to 40% (reduction of 10 percentage points), with the mean of the guesses reducing to 42.8% (a reduction of 6.0 percentage points). The distribution of guesses was narrower and more accurate after watching the memo lecture, with 75th and 25th percentiles of 50% and 35%. Despite the more accurate guesses after seeing the model answers, these guesses were still overly optimistic - The actual marks were still substantially lower than the estimates. The difference between the initial guesses and the achieved marks for A1 were: 17 percentage points for the median (14.3 percentage points for the mean). After watching the memo lecture, the difference between the guesses and the median achieved marks reduced to 7 percentage points (8.2% for the mean). We reassess the accuracy of their guesses for A2, following the intervention described next. Evaluative judgement inter vention An analysis of claimed student engagement with the online lecture material, practice resources and forums revealed that while around 97% engaged with the resources, only 38% used the Q&A forums. No student made use of the option to consult the lecturer. In other words, the majority of students did not engage in active, socially -mediated learning opportunities. Given these observations, as well as the poor performance and perception disjunctures during the A1 assessment rounds, a set of Q&A lectures was introduced using a flipped -classroom approach. The Q&A lectures were set up to ensure engagement with the content in a way that would mimic in -person and individual lecturer engagement while also emulating a question being asked during a lecture. The lectures were scheduled ad-hoc, anything from an hour to a day in advance and after hours (e.g. 9pm on a Friday). Students were invited generally to ask questions, but specifically to share problems they struggled with in the tutorials (theoretical problems such as a Boolean algebra derivation), or problems encountered during the simulated practice sessions. Students were encouraged to all turn on their cameras to make the encounter more human, especially after a year of on-and -off lockdowns. When a student asked a question, the lecturer did not help on their own device, but rather asked the student to share their desktop or webcam with the problem (written on paper or simulated on the application) with the whole class. The lecturer would then start asking probing questions to the Figure 1: Students’ assessment of sufficiency of online lectures and paper difficulty. 68 67 69 58 457481 0102030405060708090 Online lectures prepared me sufficientlyPaper was difficult After A1 After A1 memo After A1 marks After A2 Figure 2: Marks achieved for 2019, marks guessed and achieved for A1 and A2 at different times. 5650 403360 50 0102030405060708090 Class of 2019 A1Guess after writing A1Guess after memo A1Actual mark A1Guess after witing A2Actual mark A2 Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Booysen and Wolff, 2021 student to lead them to realisation. This method is similar to what would happen to in-person engagement, with the added advantage that the engagement scales to the rest of the students who joined in real -time (34%) or watched the recording (88%). In this approach, the rest of the class could also observe and engage with (1) the particular problem step with which a student was struggling, (2) the recognition of peer misconceptions, (3) the advice from the lecturer, (4) the modelling of required approaches and solutions. Although they may not have been struggling with the same steps, peer observation facilitates both cognitive conceptual alignment and confidence building (Houghton et al., 2013) . In other words, this encouraged other s tudents to come forward with their problems, thereby cascading to the most common problems students faced collectively. The class’ performance was substantially better for A2 than for A1: the median mark was 50% with 75th and 25th percentiles of 63% and 39%. Encouragingly, the difference between the initial guesses after writing and the achieved marks for A2 were less: 10 percentage points for the median and 9.6 percentage points for mean. This indicates that the students fared better at adjudicating their performance. In the fourth and final survey, students were also asked “After A1 and memo, the project, and the Q&A lectures, I had a better idea of what was expected of me” in a final question. The resounding response was affirmative from 84% of the respondents. Developing evaluative judgement Using the A1 assessment as a mediation device, students were effectively encouraged to ‘reflect-on-action’ (Schon, 1983) by considering how they had prepared for the assessment and how they thought they had perform ed. The disjuncture between their perceived and actual self-efficacy was addressed through a reflection-in -action strategy, where examples of course material were actively interrogated by students themselves in the online, flipped-classroom Q&A lecture ses sions. The first obvious benefit of this approach is the opportunity for practical, active, peer learning using mediating examples (student screen-shares). The less obvious benefit of the approach is the opportunity to overcome the broken feedback mechanis m, by which the lecturer (reflecting in action) becomes aware of the range of student ZPDs during the live Q&A sessions. While this form of online engagement may appear self-explanatory, it is important to note that in large class, resource-constrained edu cation environments (predominant in the global South), there are seldom opportunities for student- centred, individual diagnostic teaching approaches. Determining a student’s ZPD is crucial for effective teaching, and more easily facilitated in in -person teaching environments where students can ask questions. The ERT era has highlighted a significant constraint to educators’ being able to identify and respond to the range of student learning needs. Reports on poor self-regulation and low digital fluency during ERT are highly concerning, given that self-regulated learning and technology self-efficacy are predictors of academic success (Wang et al., 2013). The survey iterations in this case study offered students the opportunity to engage in Kolb’s (1984) ex periential learning cycle, where students could reflect on the experience of A1 (as well as their own perceptions of how they performed) and then actively engage in the preparation for A2. The shift in performance perception following A2 suggests a better alignment between perception and expectations. This, we argue, suggests the development of evaluative judgement through practical experience, which Bandura (1977) links to self- efficacy as a result of students experiencing the consequences of their own behaviour. In other words, being asked how they thought they would perform before and after receiving actual marks encouraged the kind of reflective practice necessary for the development of evaluative judgement. For the educator, the reflection-on -action cyc le (A1 survey iterations) and the desire to intervene productively led to the observation that the current CA model – which focuses on the relationship between intended learning outcomes, appropriate learning activities and aligned Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Booysen and Wolff, 2021 assessment - does not ex plicitly acknowledge the student position, nor the disjuncture between student perceptions and lecturer expectations. If appropriate learning activities are designed to enable students to achieve course outcomes (which are evaluated through assessment activities), then what is appropriate for whom? Observations of student performance and expectations during the ERT period (including cases beyond this study) suggest a need to develop a contextually nuanced CA framework which includes the curriculum and stakeholders. The disjunctures between different stakeholder interpretations of criteria and expectations calls for (1) a recognition of the context and (2) the need to make expectations explicit, since these impact on student interpretation of intended learning outcomes. Context implies both the available resources as well as the students themselves. In order to design appropriate teaching activities, it is essential that educators consider their students from a socio -cultural context, what they bring with them into the learning space, including their perceptions and expectations. It is only through explicit iterative practice that better alignment can be achieved between activities and intended outcomes , leading to improved evaluative judgement. This holds for both educator and student. Conclusion The Covid- 19 emergency remote teaching (ERT) phase has raised the question of the disjuncture between engineering student perceptions, assessment performance and interpretation of course expectations. Given the importance of developing evaluative judgement as a future professional engineer, the observations of a 2nd-year electrical engineering cohort at a South African institution offered the opportunity to intervene. An iterative sequence of surveys enabled students to reflect on perceptions of their practices in relation to course requirements and the mid-semester assessment. The initial gaps in perceived versus actual performance led to a reflection -in-action intervention whereby students shared and discussed their particular challenges in three online, recorded, Q&A sessions. The initial survey experience and intervention appeared to enable greater alignment between student perceptions and actual performance, as was evident in the final semester assessment. The constructive alignment framework, which provides the basis for much of outcomes -based education, does not explicitly include alignment to student abilities or perceptions. The student context is possibly implied in the descriptor ‘appropriate’ in relation to ‘l earning activities’. We propose further work on developing a CA model which includes the educator's role and expectations in relation to being able to determine ‘appropriate’ teaching activities based on a better understanding of the student context. Furthermore, this alignment in conjunction with the recognition that assessments themselves are invaluable mediating artefacts, can enable the bridging of evaluative-judgement gaps, given that we need to produce critically -thinking, confident, and capable graduates.