title: An invalidation test for predictive models

authors: Bardsley, W. Earl

subjects: model invalidation

abstract: The standard means of establishing predictive ability in hydrological models is by finding how well predictions match independent validation data. This matching may not be particularly good in some situations such as seasonal flow forecasting and the question arises as to whether a given model has any predictive capacity. A model-independent significance test of the presence of predictive ability is proposed through random permutations of the predicted values. The null hypothesis of no model predictive ability is accepted if there is a sufficiently high probability that a random reordering of the predicted values will yield a better fit to the validation data. The test can achieve significance even with poor model predictions and its value is for invalidating bad models rather than verifying good models as suitable for application. Some preliminary applications suggest that test outcomes will often be similar at the 0.05 level for standard fit measures using absolute or squared residuals. In addition to hydrological application, the test may also find use as a base quality control measure for predictive models generally.

text: An invalidation test for predictive models The standard means of establishing predictive ability in hydrological models is by finding how well predictions match independent validation data. This matching may not be particularly good in some situations such as seasonal flow forecasting and the question arises as to whether a given model has any predictive capacity. A model-independent significance test of the presence of predictive ability is proposed through random permutations of the predicted values. The null hypothesis of no model predictive ability is accepted if there is a sufficiently high probability that a random reordering of the predicted values will yield a better fit to the validation data. The test can achieve significance even with poor model predictions and its value is for invalidating bad models rather than verifying good models as suitable for application. Some preliminary applications suggest that test outcomes will often be similar at the 0.05 level for standard fit measures using absolute or squared residuals. In addition to hydrological application, the test may also find use as a base quality control measure for predictive models generally.

year: 2007

title: The effects of mask-wearing and teleneuropsychological assessment on people's memory performance on neuropsychological tests

authors: Rich, Tiffany

abstract: In the wake of the Coronavirus disease (COVID-19), the standardised conditions of neuropsychological assessments were greatly impacted to prevent the spread of COVID-19. Neuropsychological assessment conditions were altered to be conducted remotely on Zoom (teleneuropsychological) or continued in-person with the use of face masks. The changes to standardised assessment conditions led to an increase in people’s worries around assessment, made it harder to hear the assessor which led people to increase their listening effort and created more ambiguous non-verbal social cues (Bottalico et al., 2020; Aguilar & Leguizamon, 2021). These additional assessment aspects may be leading to increased pressure on people’s cognitive load by taking up more of their cognitive resources and attention, then leaving less cognitive resources to focus on the assessment. Moreover, memory tests require high rates of attention and focus to accurately encode, manipulate and retrieve information that is presented. However, the added pressure on cognitive load leaves less available cognitive resources to focus on the memory tests, which in turn may influence people’s performance on these tests. When cognitive load is overloaded, people have issues with attention, encoding and later retrieving or remembering that same information (Lezak et al., 2004). To determine whether the changes to assessment conditions (test environment and mask-wearing) impact people’s memory performance, we conducted a 2x2 mixed methods design (between-subjects factor: test environment (zoom/in-person), within-subjects factor: mask-wearing (mask/no mask). Our sample consisted of 64 participants, aged 18-59, with 43 females, 20 males and 1 participant who preferred to self-describe. Participants were included based on a screening criteria and were required to complete a range of neuropsychological tests for two sessions, either via Zoom or in-person, 1-2 weeks apart. All participants completed one session with the assessor wearing a mask and one session without. Participants scores were compared on the Verbal Learning, Digit Span and Verbal Fluency subtests between the test environments (Zoom/In-person) and mask-wearing (Mask/no mask) variables, to assess their impacts on people’s memory functioning. Results showed no significant differences between participants’ performance on these tests between the in-person and teleneuropsychological assessments or mask-wearing compared to no mask. A statistically significant but not clinically significant interaction effect was found on Verbal Learning trial 5, where participants performed the worst in the Zoom condition when the assessor was wearing a mask. This result was not clinically significant and overall, our results suggest teleneuropsychological assessment and mask wearing during assessment do not impact people’s memory performance and, therefore, are valid and reliable methods of assessment in a population aged 18-59 years with no cognitive impairments. However, we did find that the alternate forms used for the Verbal Learning subtest were not equivalent. Participants’ performed statistically significantly worse and recalled fewer words on the Crawford Verbal Learning test compared to the Rey Auditory Verbal Learning Test (RAVLT). This result was not clinically significant but is a clinically meaningful finding for neuropsychologists in New Zealand to consider when administering alternate forms of the RAVLT test.

text: The effects of mask-wearing and teleneuropsychological assessment on people's memory performance on neuropsychological tests In the wake of the Coronavirus disease (COVID-19), the standardised conditions of neuropsychological assessments were greatly impacted to prevent the spread of COVID-19. Neuropsychological assessment conditions were altered to be conducted remotely on Zoom (teleneuropsychological) or continued in-person with the use of face masks. The changes to standardised assessment conditions led to an increase in people’s worries around assessment, made it harder to hear the assessor which led people to increase their listening effort and created more ambiguous non-verbal social cues (Bottalico et al., 2020; Aguilar & Leguizamon, 2021). These additional assessment aspects may be leading to increased pressure on people’s cognitive load by taking up more of their cognitive resources and attention, then leaving less cognitive resources to focus on the assessment. Moreover, memory tests require high rates of attention and focus to accurately encode, manipulate and retrieve information that is presented. However, the added pressure on cognitive load leaves less available cognitive resources to focus on the memory tests, which in turn may influence people’s performance on these tests. When cognitive load is overloaded, people have issues with attention, encoding and later retrieving or remembering that same information (Lezak et al., 2004). To determine whether the changes to assessment conditions (test environment and mask-wearing) impact people’s memory performance, we conducted a 2x2 mixed methods design (between-subjects factor: test environment (zoom/in-person), within-subjects factor: mask-wearing (mask/no mask). Our sample consisted of 64 participants, aged 18-59, with 43 females, 20 males and 1 participant who preferred to self-describe. Participants were included based on a screening criteria and were required to complete a range of neuropsychological tests for two sessions, either via Zoom or in-person, 1-2 weeks apart. All participants completed one session with the assessor wearing a mask and one session without. Participants scores were compared on the Verbal Learning, Digit Span and Verbal Fluency subtests between the test environments (Zoom/In-person) and mask-wearing (Mask/no mask) variables, to assess their impacts on people’s memory functioning. Results showed no significant differences between participants’ performance on these tests between the in-person and teleneuropsychological assessments or mask-wearing compared to no mask. A statistically significant but not clinically significant interaction effect was found on Verbal Learning trial 5, where participants performed the worst in the Zoom condition when the assessor was wearing a mask. This result was not clinically significant and overall, our results suggest teleneuropsychological assessment and mask wearing during assessment do not impact people’s memory performance and, therefore, are valid and reliable methods of assessment in a population aged 18-59 years with no cognitive impairments. However, we did find that the alternate forms used for the Verbal Learning subtest were not equivalent. Participants’ performed statistically significantly worse and recalled fewer words on the Crawford Verbal Learning test compared to the Rey Auditory Verbal Learning Test (RAVLT). This result was not clinically significant but is a clinically meaningful finding for neuropsychologists in New Zealand to consider when administering alternate forms of the RAVLT test.

year: 2023

title: Tukua Kia Rere! Maori Adult Students Succeeding at University

authors: Williams, Daphne Christina (Tina)

subjects: Maori students in higher education

abstract: A considerable amount of research has focussed on Māori underachievement in mainstream education. There is, however, much less research that celebrates Māori academic success. Even less has been written about the experiences of adult Māori students who return to study long after leaving school and go on to become academically successful in higher education. This study contributes to the growing body of knowledge regarding Māori academic success at university. It sought to explore the experiences of sixteen adult Māori students who entered university via special admission and went on to attain undergraduate degrees. The study examined the challenges that these individual’s experienced on their way to becoming academically successful and identified the main factors that helped them to achieve educational success. An overarching Kaupapa Māori theoretical framework was employed in this study. A qualitative research methodology was implemented and Kaupapa Māori research principles informed both the research processes and the interpretation of research outcomes. The findings suggest four major factors contributed to the success of the participants in this study: a strong determination to succeed (whāia te iti kahurangi), the extended family (whānau), strong social support networks with peers and faculty (whakawhanaungatanga) and Te Ao Māori (the Māori World). However, some of these same factors were also found to hinder academic success in the university setting. The Manu Tukutuku (kite) model was developed directly from the research findings. In addition to providing a theoretical frame for understanding the experiences of Māori special admission students at university, this model also provides a useful ‘toolkit’ for those wanting to support Māori students within the university context. The implications of this study suggest that New Zealand Universities can better support Māori special admission students by being more inclusive of their cultural capital, by eliminating racism and discrimination in the university environment and by fostering better relationships between staff and students. Creating opportunities for Māori students to develop social networks at University was also identified as important.

text: Tukua Kia Rere! Maori Adult Students Succeeding at University A considerable amount of research has focussed on Māori underachievement in mainstream education. There is, however, much less research that celebrates Māori academic success. Even less has been written about the experiences of adult Māori students who return to study long after leaving school and go on to become academically successful in higher education. This study contributes to the growing body of knowledge regarding Māori academic success at university. It sought to explore the experiences of sixteen adult Māori students who entered university via special admission and went on to attain undergraduate degrees. The study examined the challenges that these individual’s experienced on their way to becoming academically successful and identified the main factors that helped them to achieve educational success. An overarching Kaupapa Māori theoretical framework was employed in this study. A qualitative research methodology was implemented and Kaupapa Māori research principles informed both the research processes and the interpretation of research outcomes. The findings suggest four major factors contributed to the success of the participants in this study: a strong determination to succeed (whāia te iti kahurangi), the extended family (whānau), strong social support networks with peers and faculty (whakawhanaungatanga) and Te Ao Māori (the Māori World). However, some of these same factors were also found to hinder academic success in the university setting. The Manu Tukutuku (kite) model was developed directly from the research findings. In addition to providing a theoretical frame for understanding the experiences of Māori special admission students at university, this model also provides a useful ‘toolkit’ for those wanting to support Māori students within the university context. The implications of this study suggest that New Zealand Universities can better support Māori special admission students by being more inclusive of their cultural capital, by eliminating racism and discrimination in the university environment and by fostering better relationships between staff and students. Creating opportunities for Māori students to develop social networks at University was also identified as important.

year: 2010