# Emulating Artistic Expressions in Robot Painting: A Stroke-Based Approach
## 3 Problem Description
### Reinforcement Learning  
The exploration problem is modeled with a Markov decision process (MDP). An MDP can be represented by \(S\), \(A\), \(P\), \(R\), and \(\gamma\), where \(A\) and \(S\) denote the actions and states, respectively. \(P\) is the transition model, which represents the probability of state transitions. Further, \(R\) represents the reward, which is calculated when action a is taken in state \(S\). \(\gamma\) is the discount factor.  
The policy gradient method will be used to obtain the optimal policy. In this paper, we used a soft actor-critic (SAC) framework [22]. The SAC framework involves two models: the actor controls the agent's behavior, while the critic evaluates the actor's action. The actor policy, \(\pi(s,a)\), maximizes the rewards. The critic function, \(Q(s,a)\), minimizes the approximation error. The two models were trained in parallel. Here, s represents the input vector that contains the information, and a refers to the next action.  
The components of the reinforcement learning setup are action space, state space, and reward design and are described as follows:  
* Action Space: An action, \(a_{t}\), is a set of parameters that determine the shape and position of the stroke at step \(t\). An agent's behavior that maps states to actions is a policy function, \(\pi\). The state, \(s_{t}\), is generated, and then the agent generates the action of the next stroke, \(a_{t}\), with the stroke parameters. The agent uses the state based on the transition function \(s_{t+1}=trans(s_{t},a_{t})\).  
Figure 2: The robotic agent network with style learning ability (several iterative cycles).  
* State Space: It consists of all possible information observed in the environment. The state of the painting system includes three parts, namely the target image, the canvas, and the number of steps.
* Reward Design: We define the reward function that allows the agent to evaluate the actions determined by the policy. The reward function is defined as the difference between the current canvas (the loss between the current canvas and the target image) and the next state (the loss between the next state and the target image). \[r(s_{t},a_{t})=L_{t}-L_{t+1}\] (1) where \(r(s_{t},a_{t})\) is the reward at step \(t\), and \(L_{t}\) represents the loss between the target image and the current canvas, \(C_{t}\). Moreover, \(L_{t+1}\) is the loss between the target image and the canvas \(C_{t+1}\). The Wasserstein generative adversarial network (WGAN) was employed to measure the difference between the generated data and the target image [23]. A GAN was used as a specific loss function in transfer learning and text modeling due to its strong ability to measure the distance between the target data and the generated data. The WGAN was developed based on the original GAN, which introduced the Wasserstein distance as a loss function between the generator and the discriminator. Unlike the loss function designed by the original GAN, the WGAN does not have the ability to reflect the degree of the feedback penalty term in the similarity between two distributions. The improvement in the WGAN greatly reduces the difficulty of training the GAN. The WGAN minimizes the Wasserstein-I distance to stabilize training. The objective of the discriminator in the WGAN is defined as follows: \[L_{discriminator}(w)=\frac{max}{w1W}E_{y\sim Y}[f_{w}(y)]-E_{x\sim X}[f_{w}(x)]\] (2) where \(E\) is the expectation for optimal coupling, and \(f_{w}\) denotes the discriminator. In addition, \(Y\) and \(X\) are the painting and target image distributions. Previous research employed GAN-based style transfer to reduce the number of strokes required for drawing sketches, resulting in shorter drawing times [24].  
This study aimed to reduce the difference between the current canvas and the target image. To achieve this, the difference in discriminator scores was used as a reward to guide the training of the actor.