# EduTVA: Metadata Schema for Educational Audiovisual Contents in Digital Television Environments
## 4 Evaluation
### Functionality Assessment  
For evaluating the functionality of the EduTVA metadata schema, we implemented Crayon, an interactive IPTV application that provides a search service of educational content that has been described with a medatada schema, such as TV-Anytime and EduTVA. Crayon enables users to input a search query, by either text or voice, compares it with the whole metadata of every educational content, and displays the results matching that query. Figure 3 shows the search results graphical interface for a query example. Since no existing IPTV applications can read the metadata structure of EduTVA, we implemented this simple search service for conducting the evaluation. Nevertheless, EduTVA can be adapted to knowledge-based applications, as discussed in Section 3.4. Further research is needed to evaluate the performance of EduTVA in such advanced applications. Note that the back-end search service in Crayon might operate in a datacasting deployment for assisting the education of people with poor Internet access at home [54; 55]. Particularly, such a deployment involves a television broadcasting station with a viable Internet connection for accessing an IPTV service, which offers a search service such as Crayon. The television station can filter educational resources from the IPTV service and broadcast the educational content via digital terrestrial television, such as DVB-T and ATSC, to reach multiple learners over a wide area [56; 57]. Furthermore, the television broadcasting station might implement a low-bandwidth return path (e.g., SMS, GSM) to realize interactivity with the IPTV search service [57], as well as broadcast multimedia content from sources other than the IPTV service. The implementation of the datacasting deployment is out of the scope of this paper. The functionality assessment then consists of validating if the search service in Crayon returns the expected results when executing different search queries to look for educational content described using both TV-Anytime and EduTVA. To measure this functionality, we adapted the OPERational Consistency (OPC) quality attribute from the ISO/IEC 25010 standard [58]. Table 2 details the OPC metric as used in this evaluation. Figure 3: Search results graphical interface in Crayon. For the television content to look for in Crayon, we employed 15 programs, most of them educational. Table 3 describes each television program, including their number of contents and segments. Note that only the program P11 presents segments; the four segments correspond to two segments of two different pieces of content. In total, Crayon incorporated 60 pieces of television content. The evaluation involved two participants. Participant \(A\) used EduTVATool (see Section 4.1) to mark up all the 60 pieces of television content with both TV-Anytime and \begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt}} \hline \hline
**Characteristic** & **Details** \\ \hline Purpose of the metric & \begin{tabular}{l} OPerational Consistency (OPC) aims at measuring how consistent are the results of the search service \\ \end{tabular} \\ \hline Method of application & \begin{tabular}{l} Counting and validating the educational content obtained on the search service when executing different queries \\ \end{tabular} \\ \hline Measurement, formula, and data element computations & \begin{tabular}{l} OPC = 1—(IR / ER); \\ \(\circ\) IR: Inconsistent Results \\ \(\circ\) ER: Expected Results \\ \end{tabular} \\ \hline Interpretation of value & \
\begin{tabular}{l} 0 \(\leq\) OPC \(\leq\) 1; the closer to 1, the better \\ \end{tabular} \\ \hline Metric scale type & Absolute \\ \hline Measure type &
\begin{tabular}{l} \(\circ\) IR units: count \\ \(\circ\) ER units: count \\ \(\circ\) OPC units: proportion \\ \end{tabular} \\ \hline Target audience & User \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Details of the OPerational Consistency (OPC) metric.