# Induced Emotion-Based Music Recommendation through Reinforcement Learning
## 4 Music Recommendation Based on Go-Explore
### The Methodology
#### 4.3.2 Step 1: Listen until Solved  
The goal is to discover high-performing trajectories in the _emotion space_, to be improved in Step 2. The result is an archive of different emotional states, named \"cells\", and trajectories to follow to reach these states. This step is organized into several _listening sessions_. The goal of each session is to find high-performing trajectories for one specific target emotional state chosen by the user. Indeed, at the beginning of the session, the user declares the state he would like to reach at the end of it, i.e., the _target emotion_ he would like to experience after the listening session. At the beginning of each session, the archive only contains the initial _emotion_ selected by the user (start state). From there, the system repeats the following steps: (i) select a cell from the current archive, (ii) explore from that cell location stochastically, i.e., recommend random music and collect feedback from the user after the listening, (iii) add new cells and their trajectory to the archive. Here, we provide details about the cell, state representation, and the reward function based on the feedback the user provided at the end of each listening. Cell and State Representation  
In order to be tractable in high-dimensional state spaces like the emotional space (see Figure 2), Step 1 of _Modify_ reduces the dimensionality of the search space into a significant low-dimensional space. Our idea is to conflate \"similar\" emotions in terms of musical features required to stimulate them in the user, in the cell representation. To this aim, in our approach, first, we have discretized the emotions space represented by the circumplex model (see Section 3.1) into a grid in which each cell is \(8\times 8\). Then, we decided that each state contains information about a specific emotion and the set of audio features that a music song should have to arouse this emotion. Selecting and Returning to Cells  
Step 1 selects a cell at each iteration. _Modify_ preferred cells _(i)_ not visited often, _(ii)_ recently used to discover a new cell, and _(iii)_ expected near undiscovered cells. _Modify_ stores the sequence of musical songs that lead to a cell to avoid added exploration. Figure 3: The emotional states are selectable in MoodifyWeb. Exploration from Cells  
Once a cell is reached, _Moodify_ explores the emotion perceived by the user for each of \(k=100\) training musical songs randomly selected from Spotify, with a 70% probability of listening to the previous music at each step. After each listening, the user selects the emotion perceived, i.e., the state reached. Then, he assigns a _score_ to the reached state, i.e., an integer value in \([0,10]\) which represents \"how much the emotion perceived after listening to the song is similar/close to a _target emotion_ established at the beginning of the session\". Finally, the audio features of such a state are updated. Formally, let \([x,y,s_{1},\ldots,s_{9}]\) be the start state, and let \([s^{\prime}_{1},\ldots,s^{\prime}_{9}]\) the audio features of the listened song. Then, if \(s_{i}=0\) then set \(s_{i}=s^{\prime}_{i^{\prime}}\) for each \(i=1,\ldots,9\). Otherwise, set \(s_{i}=\frac{s_{i}+s^{\prime}_{i}}{2}\), for each \(i=1,\ldots,9\). Exploration can be terminated at the end of listening to the selected \(k\) training songs limit for exploration or when the user stops the exploration/listening session. How to Update the Archive  
During the exploration of a cell, the archive can be updated in two cases. First, the agent explores a cell not contained in the archive. In this case, details about such a cell are added to the archive, together with some related information: (i) the full \"trajectory\" (both musical and emotional), in terms of a sequence of state-vectors, to follow for reaching that cell from the starting state; (ii) the current environment state; (iii) the _trajectory score_; (iv) the trajectory length in terms of the number of listened songs.