## 3 Materials and Methods
### Technological Implementation: Indexing and Retrieval Process  
Once the quality model was defined, a software system, see Figure 2, was designed to provide an implementation to the required set of metrics and to automatically gather information, calculate quality and the index, and retrieve scientific documents. For the indexing and retrieval process, we used the ElasticSearch platform, an open-source text search and analysis server that offers stable and reliable real-time retrieval services [51]. Additionally, ElasticSearch facilitates high-speed document stream processing and indexing [52]. Furthermore, ElasticSearch utilizes the Query DSL (Domain-Specific Language) for information retrieval based on the JSON format, enabling the definition of each search parameter. ElasticSearch consistently provides real-time retrieval services [53]. In this context, inverted indexes were employed as a fragmentation technique due to their high scalability, which enhances search speed through a distributed architecture. To evaluate the machine-processable quality of scientific documents, an algorithm (source code available at [[https://github.com/ityans/Algorithm-Dataset](https://github.com/ityans/Algorithm-Dataset)]([https://github.com/ityans/Algorithm-Dataset](https://github.com/ityans/Algorithm-Dataset)) accessed on 4 July 2023) was implemented to automatically calculate the values of the metrics. The objective is to assess the quality of machine-processable content for the purpose of indexing and retrieving information, with a particular emphasis on its document structure rather than the content per se. This methodology enables the algorithm to carry out the process more efficiently. For this purpose, natural language processing techniques were implemented such as an automatic text classification, extraction, and analysis, as well as the reuse of the Crossref algorithm through which the metadata of the collection of documents are automatically extracted by connecting the API with the different scientific databases where the documents are hosted. This is possible thanks to the services and features provided \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Indicator** & **Metrics** & **Normalized** & **Weight** \\ \hline \multirow{3}{*}{Accessibility w = 0.138} & Visibility & 0.333 & 0.1062 \\ \cline{2-4}  & Metadata & 0.452 & 0.1442 \\ \cline{2-4}  & Editorial policy & 0.215 & 0.0686 \\ \hline \multirow{3}{*}{Content w = 0.319} & Compressibility & 0.437 & 0.0603 \\ \cline{2-4}  & Readability & 0.119 & 0.0164 \\ \cline{2-4}  & Content structure & 0.444 & 0.0613 \\ \hline \multirow{3}{*}{Reproducibility w = 0.543} & Algorithms/source & 0.4280 & 0.2324 \\ \cline{2-4}  & Equations/models & 0.1340 & 0.0728 \\ \cline{1-1} \cline{2-4}  & Raw data & 0.4020 & 0.2183 \\ \cline{1-1} \cline{2-4}  & Processed data & 0.0360 & 0.0195 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Indicator weights and metrics. \begin{table}
\begin{tabular}{c c c c} \hline \hline  & \multicolumn{3}{c}{**Decision Hierarchy**} \\ \hline
**Metrics** & **Level 2** & **Global Priorities** & **Rank** \\ \hline \multirow{3}{*}{Accessibility} & Visibility & 33.3\% & 2 \\  & Metadata & 45.2\% & 1 \\  & Editorial policy & 21.5\% & 3 \\ \hline \multirow{3}{*}{Content} & Compressibility & 43.7\% & 2 \\  & Readability & 11.9\% & 3 \\  & Content structure & 44.4\% & 1 \\ \hline \multirow{3}{*}{Reproducibility} & Algorithms/source & 42.8\% & 1 \\  & Equations/theorems & 13.4\% & 3 \\ \cline{1-1}  & Raw data & 40.2\% & 2 \\ \cline{1-1}  & Processed data & 3.6\% & 4 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Hierarchy with consolidated priorities level 2.  
with Crossref, which are based on the principles of 20/20 metadata [52], which issues a response in the JSON format with the information of each document. This algorithm consists of three modules:  
Accessibility module: In which the level 2 metrics corresponding to visibility were evaluated, comprising 7 indicators; the set of metadata contains 14 elements and the editorial policies are with 7.