## 2 Protocol Description
### Algorithmic Development and Technical Validation  
This phase is devoted to the design, implementation, and technical evaluation of the algorithms and systems in charge of the automatic analysis of the recorded sequences to estimate the metrics and inferences devised in previous phases. It also includes the generation and evaluation of the relevant inferences from the automatic analysis. The associated flowchart is shown in Figure 3.  
The development and evaluation will require an initial compilation of existing sets of labeled data already published in the literature and available for research purposes. There are many datasets available for body analysis tasks that are may not be directly related to the final system objectives but are general enough to be used for pre-training the models that will be used and developed in different tasks (for example, body pose estimation and tracking, recognition of human activities, etc.).  
A crucial task will be the characterization of human activity at body and face level, with the aim of exploiting and improving existing human activity characterization systems to meet the new challenges posed by the proposed system (EYEFUL in our case). It will address the development of robust algorithms and the use of the different types of sensors (as previously defined), for the extraction of different data. The process will include an extensive work in the fine-grained analysis and characterization of human activities. Algorithmic solutions based on deep learning will be designed, implemented, and tested using existing datasets, and refined with data extracted from recorded sessions in the space where the people will carry out the selected ADLs.  
Once the set of metrics has been defined and the algorithms for their automatic extraction have been developed, their integration will be addressed in two directions.  
* The first one by integrating them into the process of assessing functionality. So, from the catalog of defined elements, we will select the most discriminative subset in order to score the functional level. Machine learning techniques are to be used in the selection of those metrics in a data driven approach. The close coordination with the clinical team will again be necessary in this effort.
* The second one, once the extracted metrics form different sensors are validated, their fusion is to be considered to improve the overall estimations (related to the users' location and pose, activity characterization and assessment, emotional state characterization, etc.).  
Finally, in the technical validation stage, the results of the algorithms will be compared with manually annotated and labeled data. As detailed above, these data will initially come from publicly available datasets and, later on, from actual recordings of subjects' test sessions within the protocol scope. Specifically, careful image annotation work will be needed for the evaluation of the defined metrics (more on this in Section 2.8). Again, a close coordination between engineers clinicians will be necessary to properly assess the evaluation results of the metrics and inferences with a clinical perspective.  
As an example of the application of this phase to the EYEFUL project, Figure 4 shows a prospective simplified schematic example of the algorithmic system modules needed for automatic human functional evaluation, showing the sensor sources, some automatic analysis modules (object detection, human action recognition, etc.), and the output information to be generated (quantitative metrics and inferences).