# Fully Open-Source Meeting Minutes Generation Tool
## 5 Evaluations
### Questionnaire Results on Usability and Effectiveness  
The usability and effectiveness of the proposed system were evaluated using a questionnaire based on the _Performance_, _Information_, _Economics_, _Control and Security_, _Efficiency_, _and Service_ (_PIECES_) framework [71]. The questionnaire was administered to 31 respondents. Each respondent first reviewed a provided presentation document, listened to the presentation recording, and executed the proposed system. Feedback was subsequently collected through the questionnaire, with the specific questions listed in Table 13.  
Each question represents a key aspect of the framework. Questions Q1, Q2, and Q5 focus on usability, such as ease of use, intuitive interface, and likelihood of recommending the system. Specifically, Q1 evaluates the _Service_ aspect, by measuring how quickly users can extract information. Q2 examines the _Control_ and _Security_ aspects, by assessing how intuitively users can manage the system. Q5 checks the _Economics_ and _Efficiency_ aspects, by gauging if users would recommend the system for similar tasks. Q3 and Q4 focus on the effectiveness of the system. Q3 assesses the _Information_ aspect by evaluating the accuracy of the audio-to-text conversion. Q4 addresses the _Performance_ aspect, reflecting the quality of summary and keyword extraction. Table 14 shows their response to the questionnaire.  
\begin{table}
\begin{tabular}{c l} \hline \hline
**Question ID** & **Questionnaire Questions** \\ \hline Q1 & The system was easy to use in extracting textual information from presentation videos. \\ Q2 & The system’s interface was intuitive and easy to navigate while correlating information between presentation videos and PowerPoint files. \\ Q3 & I am satisfied with the accuracy of the system in converting audio to text and correlating textual information from both presentation videos and PowerPoint files. \\ Q4 & I am satisfied with the quality of the generated summary and the extracted keywords. \\ Q5 & I would recommend this system to others for similar tasks requiring information correlation between presentation videos and PowerPoint files. \\ \hline \hline \end{tabular}
\end{table}
Table 13: Questionnaire questions on usability and effectiveness.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Question ID** & \
\begin{tabular}{c} **Strongly** \\ **Disagree** \\ \end{tabular} & **Disagree** & **Neutral** & **Agree** &
\begin{tabular}{c} **Strongly** \\ **Agree** \\ \end{tabular} \\ \hline Q1 & 3 & 0 & 6 & 14 & 8 \\ Q2 & 0 & 1 & 9 & 19 & 2 \\ Q3 & 0 & 0 & 11 & 10 & 10 \\ Q4 & 0 & 3 & 16 & 4 & 8 \\ Q5 & 2 & 4 & 3 & 14 & 8 \\ \hline \hline \end{tabular}
\end{table}
Table 14: The answers to the questions in the questionnaire.  
As shown in Table 14, the system received positive feedback on ease of use, interface intuitiveness, and recommendation likelihood. Most users found the system easy to navigate. Twenty-two respondents agreed that it was simple to extract information from videos, and twenty-one appreciated its intuitive design. These findings align with the performance of the _Whisper_ model, which achieved a low _WER_. However, mixed responses on the summary and keyword quality, as indicated by the responses to Q4, suggest that while the _BART_ model and _RaKUn2_ performed well, there is room for improvement. The system accurately aligned extracted information with presentation slides, as indicated by the responses to Q2, utilizing _PaddleOCR_ and _regex_. Overall, the system is user-friendly, accurate, and effective, with potential areas for enhancement.