## 2 The Skeptical Learning Algorithm  
Figure 1 depicts a high-level view of the functionalities of the SkeL main algorithm, as presented in Algorithm 1. On the right side, a set of \(n\) sensors \(S_{1},\ \dots,\ S_{n}\) sense the world and produce a set of streams \(\{x_{t}\}_{S_{i}}\), where \(x_{t}\) is the value collected by the sensor \(S_{i}\) at time \(t\). The function _sensorReading_ generates \(X_{t}\), which represents the set of all of the sensor data streams at time \(t\). On the left side, the user is asked to provide a label \(y_{t}\) as his/her annotation to a certain property \(P_{j}\) at time \(t\). The function _askUser_ generates the stream of labels \(Y_{t}\). All these functionalities are implemented as an APP, called _i-Log_[11], which can be downloaded from the _PlayStore_. The sensor data and user's labels are used to train a machine learning predictor PRED. PRED consists of a repertoire of \(m\) learning algorithms \(f_{1},\dots,f_{m}\), each taking a stream of data \(\mathbf{x}_{t}\) as an input and producing a score \(f_{k}(\mathbf{x}_{t},y)\) for the labels, which are possible values of a certain property \(P_{j}\). Similarly to the user, PRED provides its internally predicted labels \(\hat{Y}\).  
Both the user's label \(y_{t}\) and the predictor's label \(\hat{y}_{t}\) are used by SkeL during the comparison phase. We can also see \(y_{t}\) as the user's perception of the world, whereas \(\hat{y}_{t}\) is the prediction generated by the machine with the knowledge learned so far from data. The conflict between the two labels happens when the predictor is not trained enough or the user provides an incorrect label. The main idea of SkeL is to check the correctness of the predicted label and the user's annotation by leveraging its prior knowledge. This step is carried out by the function _solveConflict_. This function first checks whether the two labels are compatible and, when a conflict arises, two situations are possible. The first one is when the predictor has low confidence. In this situation, the user's label is selected and the predictor is retrained with the new data. The second situation is when the predictor's confidence is high enough: _solveConflict_ calls function _challengeUser_ to have another round of the interaction by asking again for the user's annotation. As we are assuming a collaborative  
Figure 1: The main components of the system.  
rather than adversarial user, his/her second annotation \(y^{*}\) is taken as the correct label and added to the training set to retrain the predictor. The predictor can improve its performance after rounds of training.  
```
1:procedureSkeL(\(\theta\))
2: init \(\mathbf{c}^{\text{u}}=1\), \(\mathbf{c}^{\text{p}}=0\)
3:whiletrainMode(\(\mathbf{c}^{\text{p}},\mathbf{c}^{\text{u}},\theta\))do
4:\(\mathbf{x}_{t}=\textsc{sensorReading}()\)
5:\(y_{t}=\textsc{askUser}()\)
6:\(\hat{y}_{t}=\textsc{Pred}(\mathbf{x}_{t})\)
7:\(\textsc{train}(\mathbf{x}_{t},y_{t})\)
8:\(\textsc{update}(\mathbf{c}^{\text{p}},\hat{y}_{t},y_{t})\)
9:whilerefineMode(\(\mathbf{c}^{\text{p}},\mathbf{c}^{\text{u}},\theta\))do
10:\(\mathbf{x}_{t}=\textsc{sensorReading}()\)
11:\(y_{t}=\textsc{askUser}()\)
12:\(\hat{y}_{t}=\textsc{Pred}(\mathbf{x}_{t})\)
13:\(\textsc{solveConflict}(\mathbf{c}^{\text{p}},\mathbf{c}^{\text{u}},\mathbf{x} _{t},\hat{y}_{t},y_{t})\)
14:whileTruedo
15:\(\mathbf{x}_{t}=\textsc{sensorReading}()\)
16:\(\hat{y}_{t}=\textsc{Pred}(\mathbf{x}_{t})\)
17:if\(\min\limits_{\hat{y}_{t}^{\prime}\in\textsc{SMSETS}(\hat{y}_{t})}\textsc{ Conf}(\mathbf{x}_{t},\hat{y}_{t}^{\prime},\mathbf{c}_{\hat{y}_{t}^{\prime}}^{ \text{p}})\leq\theta\)then
18:\(y_{t}=\textsc{askUser}()\)
19:\(\textsc{solveConflict}(\mathbf{c}^{\text{p}},\mathbf{c}^{\text{u}},\mathbf{x} _{t},\hat{y}_{t},y_{t})\)
```  
**Algorithm 1** Skeptical Learning (SkeL)  
The following subsections provide a detailed view of the main components implemented in the back-end.