## 3 Contextual-Bandit-Based Reminder Care System
### Conducting Recommendations
#### 3.3.2 Method  
The problem is formulated based on a contextual bandit approach to tackle the dynamicity of human activity patterns and to recommend the correct item without having to wait for the user's feedback. Contextual bandit provides a learning model based on context. Three kinds of context are extracted at this stage:  
* **Past activities context (PAC)**: Note that each activity is desired to have a different pattern; thus, for each activity, the system extracts the path/sequences of items used from the past records (recorded in the log file) as a type of context. The observed paths of each activity are then stored in a memory based on which the agent can decide an item to be recommended at a specific situation. * **Current activity Context (CAC)**: The contexts on the current states are extracted from the received data obtained from the previous two stages. For example, when the system receives that the user needs a prompt for preparing coffee, the context of the current activity (locations, previous items, user position and time) will be extracted. * **Item context (IC)**: This essentially concerns information about items, such as determining to which activity an item belongs, how long such an item can be in use, and how many times such items are needed by the user for the current activity. For example, a coffee machine as an item can be used for the activity of 'preparing coffee', where it can be used for around 2 min each time. The contextual features of each session of the activities are received by the agent as input (Algorithm 1). The CB combined three main components: an environment, which represents the context of the user's activity \(x\in X\), an agent G, which chooses an action \(a\in A\), which is represented as an item in our system (notice that the common name in CB is Learner but we call it an agent in our case) based on the received context and a \begin{table}
\begin{tabular}{c c} \hline \hline
**Notation** & **Explanation** \\ \hline \(G\) & Agent \\ \(x\), \(X\) & Context, set of context \\ \(a\) & Action(item) \\ \(r\) & Reward \\ \(A\) & Set of items \\ \(M\) & Memory \\ \(S\) & State \\ \(Miu\) & Minor update \\ \(Mju\) & Major update \\ \(T_{r}\) & Reward Delay Period \\ \(\prod\) & Policies \\ \(SV\) & State value of each sensor \\ \hline \hline \end{tabular}
\end{table} 
Table 1: System notations. reward \(r\in[0,1]\), which the agent aims to maximize by recommending the correct action at each round t= 1, 2,...,T. We calculate the expected reward after each recommendations as follows:  
\[r(a)=\frac{\sum_{t=0}^{T_{r}}SV_{a,t}}{\sum_{a=0}^{A}\sum_{t=0}^{T_{r}}SV_{a,t}} \tag{1}\]  
where \(T_{r}\) is the Reward Delay Period (this is explained below) and \(SV\) depicts the state value of each sensor at each time step \(t\). The reward function for automatically obtaining the feedback without waiting for the user is formulated. As shown in Figure 4, the calculated reward gives a non-correct recommended item (the coffee machine). Consequently, the system considers it for updating. Two types of updates are carried out to keep the system in constant interactive recommendation, minor update and major update. In minor update \(Miu\), the system is updated after the agent receives the reward where the system compares the recommended item with the item already used by the user instead of it. Such item is then updated with the newly used item to keep tracking the remaining items of the activity. ```
1:Initialize the capacity of storage memory \(M\)
2:Initialize a timer= \(24h\)
3:for session\(v_{t}\in V\)do
4: Observe state \(s_{t}\)
5: Extract \(x_{t}\), where \(x_{t}=PAC,CAC,IC\)
6: Execute action following set of policies \(\prod\)
7: waiting for \(T_{r}\)
8: Compute \(r(a)\) according Equation (1)
9: Minimize \(R\) according Equation (2)
10: update \(Miu\)
11: Put \(x_{t},r_{t},a\) into \(M\)
12: Update \(Miu\)
13:endfor
14:return
```  
**Algorithm 1** Our procedure to recommended a correct item for user's activity.