Year: 2019

Abstract:
We present a novel approach to train a natural media painting using reinforcement learning. Given a reference image, our formulation is based on stroke-based rendering that imitates human drawing and can be learned from scratch without supervision. Our painting agent computes a sequence of actions that represent the primitive painting strokes. In order to ensure that the generated policy is predictable and controllable, we use a constrained learning method and train the painting agent using the environment model and follows the commands encoded in an observation. We have applied our approach on many benchmarks and our results demonstrate that our constrained agent can handle different painting media and different constraints in the action space to collaborate with humans or other agents.

heading: 1 INTRODUCTION
text: Throughout human history, painting has been an essential element of artistic creation. There are many diverse and complex artistic domains with various styles such as watercolor, oil painting, sketching, and so on. As image processing and computer graphics have advanced, there has been a considerable effort to simulate these styles using non-photorealistic rendering (NPR) techniques (Kumar et al. (2019)). Hertzmann (1998); Winkenbach & Salesin (1996) generate compelling results using stroke-based rendering. However, most prior methods in NPR are engineered for a specific application or task, and cannot easily adapt to new styles or medium.
Recent developments in machine learning have resulted in significant advancements in computer vision and computer graphics, including computer-based painting systems. Many visual generative methods based on generative adversarial networks (Goodfellow et al. (2014)) as Zhu et al. (2017); Zhou et al. (2018); Huang et al. (2018); Karras et al. (2017); Sangkloy et al. (2017) have demonstrated promising results. Many of these machine learning methods have also been applied to stroke-based rendering tasks, including modeling the brush (Xie et al. (2012); Zheng et al. (2019)), generating brushstroke paintings in an artist’s style (Xie et al. (2015)), reconstructing drawings for specific paintings styles (Tang et al. (2018)), and constructing stroke-based drawings (Ha & Eck (2017a); Zhou et al. (2018); Huang et al. (2019); Jia et al. (2019a)).
In this paper, we focus on a more general and challenging problem of training a natural media painting agent for interactive applications. Given a reference image, our goal is to develop a stroke-based rendering approach that can imitate the human drawing or strokes used in generating the image. A key challenge is to develop a method that can learn from scratch without any supervision. In this regard, we present a technique that can handle all inputs and train an agent to manipulate natural painting media such as charcoal, pencil, watercolor, and so on. We build a model-based natural media environment using deep CNN and train a natural media painting agent using model-based reinforcement learning. In order to introduce controls to the agents for interactive applications, we use a constraint representation along with a different framework for training and use the constrained painting agent. These constraints enable the agent to interact with a human or other agents and generate various styles without retraining the model. The novel contributions of our work include:
• A method to train an agent that produces a stream of actions subject to constraint for each action. These constraints can include restricting the start location, stroke width, color, and other stroke parameters.
• A method to roll out constrained agents so the user can produce new stylistic effects interactively or automatically, as the agent is painting by modulating the action stream. • By incorporate coarse-to-fine strategy, our painting agents can generate high-resolution
stylized images using various constraints and paintbrush configurations.
We evaluate our algorithm on different paintbrush configurations and datasets to highlights its benefits over prior reinforcement learning based methods. We also employ differing constraint settings to validate our constrained agents and produce new stylistic effects with a single trained model.

heading: 2 RELATED WORK
text: In this paper, we focus on the stroke-based rendering problem, which renders the reference image with brush strokes. In contrast to image analogy approaches (Hertzmann et al. (2001); Gatys et al. (2015); Vondrick et al. (2016); Zhu et al. (2017); Karras et al. (2017); Sangkloy et al. (2017); Li et al. (2017; 2018)), stroke-based approaches can generate intermediate painting states for interactive purposes. They can also be deployed in various painting environment to generate different artistic effects.
One category of approaches for stroke-based rendering uses a heuristics-based method to guide the agent while painting. Notable examples include the method in Winkenbach & Salesin (1996), which can produce stylized illustrations, and the method in Hertzmann (1998), which can reproduce a colorful painting in different styles. However, it is difficult to extend these methods to different styles of painting because of the hand-engineered features.
Another category of approaches uses machine learning techniques to learn the policy. Compared with predefined policies, these methods use machine learning techniques to learn the painting policy, which enables the agent to generate a more natural result. Ha & Eck (2017b) train an RNN to learn the latent space of the sketch data and generate the paintings in this latent space, which requires the paired dataset for training.
Other approaches use deep reinforcement learning to learn the policy without supervised data. There have been a few attempts to tackle related problems in this domain. Xie et al. (2012; 2015; 2013) propose a series of works to simulate strokes using reinforcement learning and inverse reinforcement learning. These prior approaches learn a policy either from reward functions or expert demonstrations. Comparing with these approaches, we use a more general setup which does not rely on rewards engineering.
Zhou et al. (2018) is based on the Deep Q network. Ganin et al. (2018) trains the discriminator and the reinforcement learning framework at the same time. However, both methods can work on either a small action space or a small observation space. Jia et al. (2019b;a) use proximal policy optimization with curriculum learning and self-supervision to gradually increase the sampled action space and the frequency of positive rewards. Zheng et al. (2019) train a differentiable painting environment model, which helps learn the painting policy with gradient-based methods. Huang et al. (2019) train a model-based RL using differentiable environment and DDPG (Lillicrap et al. (2015)). This approach is limited by the OpenCV based renderer and the uncontrollable agent.

heading: 3 OVERVIEW
text: An overview of our approach is given in Fig. 1 This includes the training phase and extracting the underlying neural network as the painting agent is used for the roll-out process. We highlight all the symbols used in the paper in Table 1.
In section 4, we present the details of natural media painting agent with a environment model for the agent to interact. Our approach is based on the training scheme described in Huang et al. (2019) and uses Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al. (2015)) to train the modelbased painting agent. For the roll-out algorithm, we apply the painting agent directly to the real environment Rr as shown in Figure 1. In section 5, we present our representation of the underlying constraints and the underlying techniques to encode and decode these constraints using reinforcement learning. We use the unconstrained agent to identify the constraints, and encode them as observations to train the constrained painting agent, as shown in Figure 3. In order roll out the
constrained agent, we replace the unconstrained agent with user defined constraints or the action of previous step.

heading: 4 NATURAL MEDIA PAINTING AGENT
text: In this section, we present our algorithm for training a natural media environment model and training a painting agent to interact with this environment model.

heading: 4.1 RENDERER
text: The renderer of our painting agent generates the corresponding canvas by the given action, which implements the blending functions and other synthetic programs. Unlike the previous learning-based painting approaches Jia et al. (2019a); Huang et al. (2019); Zheng et al. (2019), we use a natural media painting renderer MyPaint (libmypaint contributors (2018)) for our experiment. Compared with self-defined painting simulators, it provides rich visual effects and sophisticated blending functions. Ganin et al. (2018) use the same environment but have not explored the various configurations of the paintbrush. Using a pre-existing environment setup helps our approach concentrate on the learning algorithm to make it easy to generalize.

heading: 4.2 ACTION REPRESENTATION
text: The action of our painting agent consists of the configurations it use to interact with the environment. For the action space, we follow the stroke definition of Ganin et al. (2018) using a quadratic Bezier curve (QBC) of three points ((x0, y0), (x1, y1), (x2, y2)). Each stroke is a 3-point curve. The pressure which affects the blending function is linear interpolated by the value of the start position (x0, y0) and the end position (x2, y2). We use one color (R,G,B) for each stroke, and the transparency of the pixels within the stroke is affected by the pressure (p0, p1). We use one brush size
r for each stroke, and the actual stroke width within the stroke can also be affected by the pressure (p0, p1). Formally, we represent the action at as:
at = (x0, y0, x1, y1, x2, y2, p0, p1, r, R,G,B) (1)
In practice, at ∈ R12. Each value is normalized to [0, 1]. The action is in a continuous space, which makes it possible to control the agent precisely.

heading: 4.3 OBSERVATION REPRESENTATION
text: The observation of our painting agent consists of information to make decisions. In our setup, the observation is represented by the reference image s∗, the current canvas st at step t, and the constraint ct. Formally, ot = (s∗, st). For the constrained agent described in section 5, ot = (s∗, st, ct). ct is a vector drawn from the sub-space of the action space. To unify the observation representation for implementation, we upsample the constraint vector as a bitmap with size of st to feed into the CNN of the policy network and the value network.

heading: 4.4 REWARD
text: Reward is a metric that enables our painting agent to measure effectiveness of the action. Aside from the reinforcement learning algorithm, the definition of the reward can be seen as a guidance for the policy π. For our problem, the goal of the painting agent is to reproduce the reference image s∗. First, we define the loss function L between the current canvas st at step t and the reference image s∗. In practice, we use l2 loss (Jia et al. (2019a)) and WGAN loss (Huang et al. (2019)) for implementation. Then, we use the difference of the losses of two continuous steps as the reward rt. We normalize rt using Eq.2, such that rt ∈ (−∞, 1].
rt = L(st−1, s ∗)− L(st, s∗) L(s0, s∗) , (2)
where L is a loss function defined as l2 or WGAN. For our reinforcement learning setting, the objective is to maximize the discounted accumulative rewards qt = ∑tmax t=1 rtγ
t, where the discount factor γ ∈ [0, 1]

heading: 4.5 ENVIRONMENT MODEL
text: For this painting environment, we use the rendering function R to represent the transition function. The action only modifies the current painting state st with the current action at as st+1 = R(at, st). Inspired by Huang et al. (2019) and Zheng et al. (2019), we model the behaviors of the real environment renderer Rr using a neural network as Rn. We use a 4-layer fully connected neural network followed by 3 convolutional neural networks.
There are two main benefits of modeling the environment. First, the reinforcement learning algorithm may have millions of steps, and it can be computationally expensive to use the real environment because of sophisticated synthetic algorithms. If we simplify the environment model, it can save training time. Second, modeling the transition function of states using neural network makes the entire framework differentiable, and the loss can be backpropagated through the renderer network Rn.
In practice, we model Rr by building a dataset consisting of paired data (at, st, st+1). To simplify the problem, we always start from a blank canvas s0 so that the Rn only need to predict the stroke image from a single action. At run-time, we synthesize the blending function by making the stroke image transparent and adding it to the current canvas st. The paired dataset {a(i), s(i)} can be collected by randomly sampling the ai and synthesising the corresponding si.
We treat different paintbrushes of MyPaint (libmypaint contributors (2018)) as separate models. It is worth noting that we use the environment model Rn for the training process, but use the real environment Rr for the roll-out process.

heading: 5 CONSTRAINED PAINTING AGENT
text: After we train the natural media painting agent, we make the agent controllable to handle interactive scenarios and be used for generating stylized images.
To achieve this goal, one straightforward way is to constrain the agent with a modified action space. However, this method is difficult to generalize to different constraints. For each different constraint c in the same subspace C in action space A, the policy needs to be retrained. For example, if we train a painting agent and constrain its stroke to width 0.8, we still need to train it again when we need an agent to output a stroke with width 0.9.
In the rest of this section, we propose a constrained painting agent that, can follow any constraint c drawn from the subspace of action space A. First, we define the constraint representation, including the definition of the constraint, the encoding function, and the decoding function. Then we introduce the training and roll-out schemes of the constrained agent.

heading: 5.1 CONSTRAINT REPRESENTATION
text: The constraint is the vector c in the constraint space C, which is the sub-space of the action space A. While fixing the vector c, the agent has to explore the subtraction of A and C as A′ by sampling a′ from A′ and concatenating with c:
a = a′ ⊕ c, a′ ∈ A′ = A− C, c ∈ C (3)
We can define C by selecting and combining dimensions in A. For example, we can constrain the color of the stroke by defining C as the color space (R,G,B). We can constrain the start position of the stroke by defining C as (x0, x1). We can constrain the stroke width of the stroke by defining C as (r). Moreover, we can use the combination of the constraints to achieve complex effects.
As shown in Figure 3, the unconstrained agent takes the reference image and the current canvas (s∗, st) as observation while the constrained agent takes the reference (s∗, st, ct) as an observation, which has an additional constraint ct at step t. ct is a vector drawn from the sub-space of the action space. To unify the observation representation for implementation, we encode the constraint vector ct as a bitmap to feed into the CNN of the policy network and the value network.
To encode the constraint ct into observation, we upsample the constraint ct as a matrix c′t which has the same size of s∗ and st, to stack them and feed into our policy network and our value network. To decode the constraint c′t into ct, we implement a downsample module within the policy network. The policy network pi can be seen as separate policy: πc and πa; πc outputs the downsampled constraint c and πa outputs the constrained action a′. Then we concatenate a′ and c to form action a.
Formally, we have ct = πc(c′t), a ′ t = πa(s ∗, st, c ′ t), and at = a ′ t ⊕ ct.

heading: 5.2 TRAINING THE CONSTRAINED PAINTING AGENT
text: After we introduce the constraint ct as part of the observation of the painting agent, we propose the corresponding training scheme. Because the constraint representation is designed for interactive purposes, we need to use either human experts or another agent to train the constrained agent.
As shown in Figure 3, we use an unconstrained agent to generate constraints by cascading them together. For each step t, the unconstrained agent takes the reference image and the current canvas
(s∗, st) as observations, and outputs the action a in action space A. Then we identify a subspace in A as constraint space C and transfer at to ct, followed by upsampling the ct as c′t, as defined in section 4. After that, the constrained agent takes the additional constraint c′t as an observation (s∗, st, c ′ t) and outputs action at concatenated by ct and a ′.

heading: 5.3 ROLL-OUT CONSTRAINED PAINTING AGENT
text: For the roll-out of the constrained painting agent, we replace the unconstrained agent from the training scheme in Figure 3 with the constraint extracted from either human command or a painting agent’s output for each step t.
When the constraint is extracted from a human command, the agent can be ordered to place a paint stroke with a given position (x0, y0), color (R,G,B), and stroke width r. When the constraint is extracted from painting agent’s output, it can be seen that we split the one agent that explores in entire action space A into two agents that explore separate action spaces A0 and A1, where A = A0 +A1.

heading: 6 RESULT

heading: 6.1 NATURAL MEDIA PAINTING ENVIRONMENT MODEL
text: For training the natural media painting environment, we treat different paintbrushes as different models. We run 50,000 iterations for each model and record the l2 loss on the validation dataset as shown in Table 2. The learning curves of the training processes are shown in Figure 4 (left).

heading: 6.2 UNCONSTRAINED PAINTING AGENT
text: We train the unconstrained painting agent using the fixed envionment model with various dataset. We use hand-written digits images (MNIST LeCun & Cortes (2010)), character images (KanjiVG Apel (2014), face images (CelebA Liu et al. (2015), object images (ImageNet Deng et al. (2009)) as train painting agents. We run 10,000 episodes for each training task. Because of the difference among the dataset, we use 5 strokes to reproduce hand-written digits images, 20 strokes to reproduce character images, 100 strokes to reproduce face and object images.
We measure the l2 loss between the reference images and reproduced images throughout the training process shown as Figure 4(middle). The reproduced results is shown as Figure 5, and the corresponding l2 loss is shown as Table 3(left).

heading: 6.3 CONSTRAINED PAINTING AGENT
text: We train the constrained painting agents using the learning parameters as unconstrained painting agents. We compute the l2 distance between the reference images and reproduce results for the training process with various constraint configurations. To control variates, we use same neural environmentRn (charcoal) and dataset (CelebA) for these experiments. We use the color (R,G,B), the stroke width r, the start position (x0, y0) and all of them (x0, y0, r, R,G,B) as constraints.
The learning curves of these constraints are shown as Figure 4(right) and the l2 loss is shown as Table 3(right). We demonstrate the constrained painting agents as Figure 2, which uses pencil as paintbrush and incorporates coarse-to-fine strategy by dividing the reference images as 4×4 patches. To constrain the color of strokes, we build a color palette by clustering the colors of the reference image. For each action a, we constrain it from a color randomly selected from the color palette. To constrain the stroke width, we use a constant stroke width for the roll-out process.

heading: 7 CONCLUSION AND FUTURE WORK
text: In this paper, we train natural media painting agents that can generate artistic paintings using various natural media, and collaborate with humans and other agents to get different visual effects. We build a model of natural media environment using deep CNN and train a natural media painting agent using model-based reinforcement learning. To introduce controls to the agents for interactive purposes, we propose constraint representation, a framework for training a constrained painting agent, and various roll-out schemes to apply the agent. We demonstrate our algorithm by applying the trained model using various paintbrushes from MyPaint and constraints set up. The experimental results show that our algorithm can reproduce reference images in multiple artistic styles.
For future work, we aim to extend the proposed algorithm by building a unified model for differing paintbrush configuration. In addition, we will train a hierarchical agent that uses a constrained agent as the low-level policy. We would like to apply our approach on other reference images and use for interactive painting systems.

heading: A APPENDIX

Review summaries:
This paper presents a reinforcement learning agent trained to interact with a painting environment in order to reproduce target images. The novel aspect of this work seems to be a version of the agent producing partial actions (i.e., portion of the action tuple is clamped to pre-specified values). This new agent receives the clamped components of the action as an additional conditioning input.