# A Taxonomy of Idea Management Tools for Supporting Front-End Innovation
## 4 Discussion
### What Are the Evaluation Criteria of Idea Management Tools?  
How useful do users consider idea management tools? Do the tools help provide a better idea management experience? Nineteen papers evaluating the performance of ideas management tools, reviewing the evaluation design, and testing the results are necessary to consider. Most of the studies are conducted by academic researchers. Fewer are designed by corporations such as Allianz [36]. Only some tools are tested in real contexts with target users [8; 13; 22; 34; 36; 40]. As we mentioned, they often use students as subjects. Table 4 displays different criteria to evaluate the idea management tools. The research team combined the requirements with the exact definition. Most standards look to justify the idea's quality and impact. Some criteria are measurable, such as quantity, satisfaction, and usability.  
Regarding quality, improvement, and appropriateness, it is necessary to involve experts or users to identify the qualitative performance and turn it into a score [51]. Novelty, quantity, and quality are the most mentioned criteria. However, measures depend on different idea management requirements, such as radical and incremental innovation. Therefore, we should further explore the effectiveness of other criteria combinations, especially in different innovation types.  
Researchers adopted an experimental comparison (\(n\) = 14, 73.7%) to evaluate the effectiveness and performed longitudinal experiments (\(n\) = 5, 26.3%). It was found that these methods map the classification of tools since the longitudinal investigation requires the mutual idea management system to test over months or even years. Furthermore, to evaluate the performance of tools, researchers need a benchmark against which to compare the performance. Researchers tend to use the usual methods to solve design problems, such as brainstorming and the K-J method. However, there are more techniques to improve the performance of idea management. [PERSON] and [PERSON] found that researchers have created a wide range of potential routes for further study, demonstrating the infinite potential of this subject [20].  
Supposing the idea management system relies on crowdsourcing, it is more technology-oriented and capable of managing a large amount of data. Crowdsourcing markets have gained popularity and show promise, facilitating theoretical and empirical research into the creation of algorithms to improve several features of these markets [61]. Still, these data are more quantitative, including satisfaction and feasibility. The convergence of ideas for further examination from a vast pool of candidate ideas of varying quality poses a significant challenge [59]. Therefore, we must decide which technique has the best performance. Future directions are related to the improvement of selecting the more popular benchmark and investigating how to enhance the idea quality in the crowdsourcing system.  
\begin{table}
\begin{tabular}{c c c} \hline
**Criteria for Evaluating Ideas** & **Frequency** & **Reference** \\ \hline Novelty & 11 & [2,7,10,13,35,37,39; 47; 48; 54; 55] \\ Quantity & 10 & [2,7,8,10,13,16; 25; 34; 47; 48] \\ Quality & 9 & [2,5,7,10,13,25; 34; 47; 51; 53] \\ Variety & 6 & [13; 16; 35; 43; 47; 48] \\ Usability & 6 & [2,7,10,11,21,54] \\ Satisfaction & 2 & [3; 6; 40] \\ Feasibility & 2 & [8; 54] \\ Improvement & 2 & [34; 42] \\ \hline \end{tabular}
\end{table}
Table 4: Frequency of criteria for evaluating ideas in the literature.