## 4 Evaluation
### Smart Metering Experiment
#### 4.1.2 Evaluation Results  
The main evaluation results are shortly overviewed in the following:  
* The detailed evaluation results of the autonomic M2M manager as a component have been presented in [65], and shortly summarized in the following. The basic mechanism relying on the IBM MAPE-K control loop model seem to work fine in the smart metering use case, however, there are still challenges related to scalability and performance;
* When considering the semantics of the data conveyed by M2M applications, ETSI has now become a part of the new ONEM2M consortium that intends to address the semantic description of M2M data. However, this is a large task that is very challenging to be handled by a single standardization, body because of multiple application domain specific views to the referred M2M data. Therefore, it is better to apply application domain specific models for the semantics of M2M application data and not to include it into the horizontal service capability layer specifications;
* Another challenge regarding the place of semantic reasoning services in the Service Capability Layer. The Service Capability Layers were designed in a way so that they only act as serving hatch for information. Certain data may be sensitive and confidential, so they are often encrypted; only emitters and final receivers of this information have the necessary credentials to decrypt it. In this case, the semantic reasoner shall be placed somewhere in the data receiver application otherwise no reasoning can be done. From the implementation point of view, ETSI M2M SCL assumes that M2M applications know all the details of the device installation and data interpretation. This is challenging for M2M application developers, and therefore, autonomic service capabilities for Device Autonomic Capability  
Figure 19: The main components of the smart metering system. (DAC), Gateway Autonomic capability (GAC), Network autonomic capability (NAC) are being created for SCL, to connect it smoothly with autonomic manager and information management, Figure 19. However, all the service capabilities are important to keep transparent for the information and let the autonomic M2M manager to enable control loop based on the M2M information;
* The use of the different Service Capability Layers reduces the complexity of integrating new devices and reduces M2M applications development time. When a new device is deployed, it's not necessary to readapt the existing systems (M2M Gateways, M2M Servers and backend applications). But the communication interfaces are quite complex to understand, an advanced knowledge of the ETSI M2M standard is necessary in order to develop applications and integrate new devices. Development APIs and frameworks shall be available for non ETSI developers;
* The detailed evaluation results of the energy efficient caching system for Constrained Application Protocol (CoAP)-HTTP proxy have been presented in [72], and shortly summarized in the following. The simulation results show that the introduction of a caching architecture has energy saving impact in the system on the system performance, since it allows reducing the transmissions inside the WSN;
* A multi-model, bi-layered framework is proposed in [73] to enhance the self-management of ETSI M2M systems. A graph-based representation built on top of the ETSI M2M standard constitutes respectively the formal and functional layers of the framework. In order to ensure inter-layers coherency, the model also comprises bi-directional communications between these two layers. The graph-based characterization allows the definition of consistency preserving reconfiguration mechanisms. On the other hand, it still possesses the functionalities granted by the standard, such as discovery protocols and machine interoperability;
* The end-to-end security architecture model has been implemented in a prototype demonstrator using Arduino devices. The initial enrolment of the devices resulting in the definition of the long term credentials was performed using an out of band channel.