title: Human evaluation of Kea, an automatic keyphrasing system.

authors: Jones, Steve

subjects: keyphrase extraction

abstract: This paper describes an evaluation of the Kea automatic keyphrase extraction algorithm. Tools that automatically identify keyphrases are desirable because document keyphrases have numerous applications in digital library systems, but are costly and time consuming to manually assign. Keyphrase extraction algorithms are usually evaluated by comparison to author-specified keywords, but this methodology has several well-known shortcomings. The results presented in this paper are based on subjective evaluations of the quality and appropriateness of keyphrases by human assessors, and make a number of contributions. First, they validate previous evaluations of Kea that rely on author keywords. Second, they show Kea's performance is comparable to that of similar systems that have been evaluated by human assessors. Finally, they justify the use of author keyphrases as a performance metric by showing that authors generally choose good keywords.

text: Human evaluation of Kea, an automatic keyphrasing system. This paper describes an evaluation of the Kea automatic keyphrase extraction algorithm. Tools that automatically identify keyphrases are desirable because document keyphrases have numerous applications in digital library systems, but are costly and time consuming to manually assign. Keyphrase extraction algorithms are usually evaluated by comparison to author-specified keywords, but this methodology has several well-known shortcomings. The results presented in this paper are based on subjective evaluations of the quality and appropriateness of keyphrases by human assessors, and make a number of contributions. First, they validate previous evaluations of Kea that rely on author keywords. Second, they show Kea's performance is comparable to that of similar systems that have been evaluated by human assessors. Finally, they justify the use of author keyphrases as a performance metric by showing that authors generally choose good keywords.

year: 2001-02-01

title: Online surveys-possibilities, pitfalls and practicalities: the experience of the TELA evaluation.

authors: Harlow, Ann

subjects: online survey

abstract: Surveys are a commonly used strategy for gaining insights into teacher perspectives. They are often seen as cost effective in terms of time and resources required, but the challenges involved in conducting and processing paper-based surveys can be considerable. Online surveys offer an alternative, particularly now that people have better access to computers and the Internet, and also since the available software has recently become more user-friendly and in some cases, free to the user. In this paper, the research process used in a large longitudinal evaluation study is discussed, where teachers increasingly opted to participate in an online survey in preference to a paper survey. The possibilities of choosing to use an online survey were attractive but they came with pitfalls. The practical strategies that proved effective in engaging teachers in an online survey process and in managing online responses are discussed and considered in the light of current research literature.

text: Online surveys-possibilities, pitfalls and practicalities: the experience of the TELA evaluation. Surveys are a commonly used strategy for gaining insights into teacher perspectives. They are often seen as cost effective in terms of time and resources required, but the challenges involved in conducting and processing paper-based surveys can be considerable. Online surveys offer an alternative, particularly now that people have better access to computers and the Internet, and also since the available software has recently become more user-friendly and in some cases, free to the user. In this paper, the research process used in a large longitudinal evaluation study is discussed, where teachers increasingly opted to participate in an online survey in preference to a paper survey. The possibilities of choosing to use an online survey were attractive but they came with pitfalls. The practical strategies that proved effective in engaging teachers in an online survey process and in managing online responses are discussed and considered in the light of current research literature.

year: 2010

title: Talk More: Student Learning Through Reciprocal Conversations

authors: Grant, Sarah

subjects: oral language

abstract: This research explored ways of improving the oral language development of Year 1 children in four low decile mainstream schools who were making limited progress in literacy. Over 82% of these students were Māori.  Due to the continued impact of past educational policies and the subsequent interruption of the intergenerational transmission of the mother tongue, many Māori children are not exposed to proficient and fluent models either of spoken English or of spoken Māori.  Such children struggle to make the English literacy progress expected of them by schools and communities. The central focus of the research was to evaluate a claim that Year 6 students (tuākana)  could make a substantial improvement in the oral language achievement of  Year 1 students (teina), through engaging with them in regular conversational contexts using the TALES (Talk, Ask, Listen, Encourage, and Say) procedures. This pedagogical approach is understood as one which was culturally responsive for these students. 

Quantitative analysis of data from three quantitative outcome measures (Record of Oral Language, Junior Oral Screening Tool, and Auditory-Vocal Association Assessment of Verbal Attainments) demonstrated that there were substantial oral language gains for the 72 teina students involved in the study. The quantitative analysis also demonstrated that these gains occurred most strongly during the phases in which the TALES procedures were being implemented by the 72 tuākana students, in accord with the multiple baseline design. Detailed qualitative analysis of a random sample of six of the 72 pairs illustrated both the effectiveness of the tuākana language interaction with the teina, and the different ways that the tuākana were able to implement the TALES procedure. Analysis of five minute probes of transcripts over six weeks from these six tuākana - teina pairs indicated that a wide range of literacy activities and conversations took place. The unique learning needs and personal learning intentions of each tuakana and teina were successfully monitored using this five minute probe procedure. Powerful reciprocal learning processes were evident in transcripts of conversations between the tuākana and the teina, and also within feedback and feed-forward meetings between the tuākana and key teachers.  

The substantial oral language gains for the teina students in this study  were achieved within learning contexts that were social and  interactive, and that embodied the principles of ako (learning and teaching roles were shared) and whakawhanaungatanga (building caring and  supportive relationships). These principles are among those that underpin pedagogies that are culturally responsive (Bishop  Glynn, 1999; Glynn, Wearmouth,  Berryman, 2006; Ladson-Billings, 1995, 2006) and transformative. The results of this research study give a clear message to mainstream non-Māori teachers, that they can make a positive and substantial difference to the learning outcomes of their Māori students.

text: Talk More: Student Learning Through Reciprocal Conversations This research explored ways of improving the oral language development of Year 1 children in four low decile mainstream schools who were making limited progress in literacy. Over 82% of these students were Māori.  Due to the continued impact of past educational policies and the subsequent interruption of the intergenerational transmission of the mother tongue, many Māori children are not exposed to proficient and fluent models either of spoken English or of spoken Māori.  Such children struggle to make the English literacy progress expected of them by schools and communities. The central focus of the research was to evaluate a claim that Year 6 students (tuākana)  could make a substantial improvement in the oral language achievement of  Year 1 students (teina), through engaging with them in regular conversational contexts using the TALES (Talk, Ask, Listen, Encourage, and Say) procedures. This pedagogical approach is understood as one which was culturally responsive for these students. 

Quantitative analysis of data from three quantitative outcome measures (Record of Oral Language, Junior Oral Screening Tool, and Auditory-Vocal Association Assessment of Verbal Attainments) demonstrated that there were substantial oral language gains for the 72 teina students involved in the study. The quantitative analysis also demonstrated that these gains occurred most strongly during the phases in which the TALES procedures were being implemented by the 72 tuākana students, in accord with the multiple baseline design. Detailed qualitative analysis of a random sample of six of the 72 pairs illustrated both the effectiveness of the tuākana language interaction with the teina, and the different ways that the tuākana were able to implement the TALES procedure. Analysis of five minute probes of transcripts over six weeks from these six tuākana - teina pairs indicated that a wide range of literacy activities and conversations took place. The unique learning needs and personal learning intentions of each tuakana and teina were successfully monitored using this five minute probe procedure. Powerful reciprocal learning processes were evident in transcripts of conversations between the tuākana and the teina, and also within feedback and feed-forward meetings between the tuākana and key teachers.  

The substantial oral language gains for the teina students in this study  were achieved within learning contexts that were social and  interactive, and that embodied the principles of ako (learning and teaching roles were shared) and whakawhanaungatanga (building caring and  supportive relationships). These principles are among those that underpin pedagogies that are culturally responsive (Bishop  Glynn, 1999; Glynn, Wearmouth,  Berryman, 2006; Ladson-Billings, 1995, 2006) and transformative. The results of this research study give a clear message to mainstream non-Māori teachers, that they can make a positive and substantial difference to the learning outcomes of their Māori students.

year: 2009