Training a Constrained Natural Media Painting Agent using Reinforcement Learning 
Authors: Biao Jia
Jonathan Brandt
Radomir Mech
Ning Xu
Byungmoon Kim
Dinesh Manocha
Venue:  (2020)

Abstract: We present a novel approach to train a natural media painting using reinforcement learning. Given a reference image,  our formulation is based on stroke-based rendering that  imitates human drawing and can be learned from scratch without supervision. Our painting agent computes a sequence of actions that represent the primitive painting strokes. In order to ensure that the generated policy is predictable and controllable, we use a constrained learning method and train the painting agent  using the environment model and follows the commands encoded in an observation. We have applied our approach on many benchmarks and our results demonstrate that our constrained agent can handle different painting media and different constraints in the action space to collaborate with humans or other agents.


Under review as a conference paper at ICLR 2020
TRAINING A CONSTRAINED NATURAL MEDIA PAINT- ING AGENT USING REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We present novel approach to train a natural media painting using reinforce- ment learning. Given a reference image, our formulation based on stroke- basee rendering that imitates human drawing and can be learned from scratch without supervision. Our painting agent computes a sequence of actions that represent the primitive painting strokes. In order to ensure that the generated policy is predictable and controllable, we use a constrained learning method and train the painting agent using the environment model and follows the commands encoded in an observation. We have applied our approach on many benchmarks and our results demonstrate that our constrained agent can handle different painting media and different constraints in the action space to collaborate with humans or other agents.
INTRODUCTION
Throughout human history, painting has been an essential element of artistic creation. There are many diverse and complex artistic domains with various styles such as watercolor, oil painting, sketching, and so on. As image processing and computer graphics have advanced, there has been considerable effort to simulate these styles using non-photorealistic rendering (NPR) techniques (Kumar et al. (2019). Hertzmann] ( (1998); Winkenbach & Salesin (1996) generate compelling results using stroke-based rendering. However. most prior methods in NPR are engineered for specific application or task, and cannot casily adapt to new styles or medium.
Recent developments in machine learning have resulted in significant advancements in computer vision and computer graphics, including computer-based painting systems. Many visual genera- tive methods based generative adversarial networks (Goodfellow al. (2014) Zhu et al. (017)): /Zou et al.) (2018): Huang et al.] (2018); Karras et al.] (2017) Sangkloy et al. (2017) have demonstrated promising results. Many of these machine learning methods have also been applied to stroke-based rendering tasks, including modeling the brush (Xie et al. (012)); Zheng et al. (2019). ) generating brushstroke paintings an artist's style (Xie et al. (2015)). reconstructing drawings for specific paintings styles (Tang et al. (2018)), and constructing stroke- drawings (Ha & Eck (0177): Zhou et al. (2018); Huang et al. (219); Jia et al. (2019a)).
In this paper, we focus on more general and challenging problem of training natural media paint- ing agent for interactive applications. Given a reference image, our goal is to develop a based rendering approach that can imitate the human drawing strokes used in generating the image. A key challenge to develop a method that can learn from scratch without any supervision. In this regard, we present technique that can handle all inputs and train an agent to manipulate natural painting media such as charcoal, pencil, watercolor, and so on. We build a model-based natural media environment using deep CNN and train natural media painting agent using model-based reinforcement learning. In order to introduce controls to the agents for interactive applications, we constraint representation along with different framework for training and use the constrained painting agent. These constraints enable the agent to interact with a human or other agents and generate various styles without retraining the model. The novel contributions of our work include:
A method to train agent that produces stream of actions subject to constraint for each action. These constraints can include restricting the start location, stroke width, color, and other stroke parameters.

---

Under review as a conference paper at ICLR 2020
A method to roll out constrained agents the user can produce stylistic effects inter- actively or automatically, as the agent is painting by modulating the action stream.
By incorporate coarse-to-fine strategy, our painting agents can generate high-resolution stylized images using various constraints and paintbrush configurations.
We evaluate our algorithm on different paintbrush configurations and datasets to highlights its ben- efits over prior reinforcement learning based methods. We also employ differing constraint settings validate our constrained agents and produce new stylistic effects with single trained model.
2 RELATED WORK
In this paper, we focus on the stroke-based rendering problem, which renders the reference image with brush strokes. In contrast to image analogy approaches (Hertzmann et al. (2001); Gatys et al. (2013); Vondrick et al. (2066); Zhu et al. (2017); Karras et al. (2017): (Snnkloyy et al. (2017), Li et al. (201) stroke-based approaches can generate intermediate painting states for interactive purposes. They can also be deployed in various painting environment to generate different artistic
effects.
One category of approaches for stroke-based rendering uses heuristics-based method to guide the agent while painting. Notable examples include the method in Wiinenbach Salesin (199),, which can produce stylized illustrations, and the method in Hertzmann (1998), which reproduce a colorful painting in different styles. However, it is difficult to extend these methods to different styles of painting because of the hand-engineered features.
Another category of approaches uses machine learning techniques to learn the policy. Compared with predefined policies, these methods machine learning techniques to learn the painting policy, which enables the agent to generate a more natural result. [a & Eck (2017b) train an RNN to learn the latent space of the sketch data and generate the paintings in this latent space, which requires the paired dataset for training.
Other approaches use deep reinforcement learning learn the policy without supervised data. There have been a few attempts to tackle related problems in this domain. Xie et al. (2012) 2015 2013) propose a series of works simulate strokes using reinforcement learning and inverse reinforcement learning. These prior approaches learn policy either from reward functions or expert demonstra- tions. Comparing with these approaches, we use a more general setup which does not rely on rewards engineering.
Zhou al. (2018) is based on the Deep Q network. Ganin et al. (2018) trains the discriminator and the reinforcement learning framework at the same time. However, both methods can work on either a small action space or a small observation space. Jia et al. (2019b a) use proximal policy optimization with curriculum learning and self-supervision to gradually increase the sampled action space and the frequency of positive rewards. Zheng et al.] (2019) train a differentiable painting environment model, which helps learn the painting policy with praseedt-based methods. |Huang eet al. (2019) train a model-based RL using differentiable environment and DDPG (Lillicrap et al. (2015)) This approach is limited by the OpenCV based renderer and the uncontrollable agent.
OVERVIEW
An overview of our approach is given in Fig. [ This includes the training phase and extracting the underlying neural network as the painting agent used for the roll-out process. We highlight all the symbols used in the paper in Table
In section 4 we present the details of natural media painting agent with a environment model for the agent to interact. Our approach is based on the training scheme described in Huang et al. (2019) and uses Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al. (2015) to train the model- based painting agent. For the roll-out algorithm, we apply the painting agent directly to the real environment R, as shown in Figure In ssecctin 3] we present our representation of the underlying constraints and the underlying techniques to encode and decode these constraints using reinforce- ment learning. We use the unconstrained agent to identify the constraints, and encode them as observations to train the constrained painting agent, as shown in Figure 3 In order roll out the
2

---

Meaning Under review as a conference paper at ICLR 2020
Table 1: Notation Summary
Symbol Meaning
Symbol
step index
81
current painting state of slep i, canvas reproduction of s* observation of step reward of step discount factor for computing the reward value function the painting policy,
8"
target painting state, reference image constraint of step
Cr
Op
action of step t, a, ae. Wa, cr] accumulated reward of step painting policy, predict by o predict by o
Ft
qt
?
VT
Rio,, 8t)
render function, render action to St
O(s",8) 81)
observation function, encode the
state and larget state
R
real renderer
R2
neural renderer
M8,8")
loss function, measuring distance between state s and objective state s'
constrained agent, we replace the unconstrained agent with user defined constraints or the action of previous step.
Critic
Reference
Reference
Actor
Painting Agent
(Painting Agent)
Rx
a
R,
Canvas
Canvas
Training the Natural Media Painting Agent (left): We use actor-critic- bastee reinforce- ment learning framework to train the painting agent. For each step, the current states of the canvas and the reference image form the observation for the policy network. Based on the observation, the painting agent predicts action for the neural renderer R2 to execute and updates the canvas accordingly. Roll-oul of the Natural Media Painting Agent (right): We extract the actor neural net- work as the painting agent for the roll-out process. We replace the neural renderer R,, with the real renderer R, to get the precise synthetic results.
Figure 1:
NATURAL MEDIA PAINTING AGENT
In this section, we present our algorithm for training a natural media environment model and training painting agent to interact with this environment model.
4.1 RENDERER
The renderer of our painting agent generates the corresponding canvas by the given action, which implements the blending functions and other synthetic programs. Unlike the previous learning-based painting approaches Jia al. 22119a) Huang et al. (2019) Zheng et al. (2019), we use natural me- dia painting renderer MyPaint (libmypaint contributors (2018) for experiment. Compared with self-defined painting simulators, provides rich visual effects and sophisticated blending functions. Ganin et al. (018) use the same environment but have explored the various configurations of the paintbrush. Using a pre-existing environment setup helps our approach concentrate on the learning algorithm to make it easy to generalize.
4.2 ACTION REPRESENTATION
The action of our painting agent consists of the configurations use to interact with the environ- ment. For the action space, follow the stroke definition of Ganin et al. (2018) using quadratic Each stroke a 3- 3-point curve. The pressure which affects the blending function is linear interpolated by the value of the start position (mo, yo) and the end position (I2, 32). We use one color (R, G, B) for each stroke, and the trans- parency of the pixels within the stroke is affected by the pressure (po, P1). We one brush size
Bezier curve (QBC) of three points ((Io- Bo), ) (21, (x2, 32)).
3

---

Under review as a conference paper at ICLR 2020
for each stroke, and the actual stroke width within the stroke also be affected by the pressure (po. P1). Formally, we represent the action ag as:
a2 = (To: you 21, U1, 22 U2. Po: P1, R, G,
(1)
In practice, a, E IR2. Each value is normalized to [o, 1]. The action is a continuous space, which makes it possible control the agent precisely.
4.3 OBSERVATION REPRESENTATION
The observation of our painting agent consists of information make decisions. In setup, the observation is represented by the reference image s", the current canvas at step t. and the constraint Ct. Formally, Or (s", st). For the constrained agent described in section 5J or (s" St, Ce). is a vector drawn from the sub-space of the action space. To unify the observation representation for implementation, we upsample the constraint vector as a bitmap with size of 8 to feed into the CNN of the policy network and the value network.
4.4 REWARD
Reward is metric that enables painting agent measure effectiveness of the action. Aside from the reinforcement learning algorithm, definition of the reward can seen as a guidance for the policy For problem, the goal of the painting agent is to reproduce the reference image s". First, we define the loss function between the current canvas s, at step t and the reference image s* In practice, implementation. Then, we use the difference of the losses of two continuous steps as the reward rt- We normalize r; using 2 such that r; E (-00,
use l2 loss (Jia et al. (2019a)) and WGAN loss (Huang et al. (2019) for
It (81-1188) T(st, 8*)
L(su, s*)
where T, is a loss function defined as l2 or WGAN. For our reinforcement learning setting, the objective is to maximize discounted accumulative rewards q: = Etm rey', where the discount factor [0,
4.5 ENVIRONMENT MODEL
For this painting environment, The action only modifies the current painting state 8, with the current action a, as 8t+1 R(a, $1 Inspired by Huang et al. (2019) and Zheng et al. (2019), model the behaviors of the real environ- ment renderer R, using a neural network as R, We use a 4-layer fully connected neural network followed by convolutional neural networks.
the rendering function R to represent the transition function.
charcoal
pencil
bulk
watercolor call giligraphy
Figure 2: Environment Model We model the painting environment MyPaintlibmypaint contributors ) with different paintbrush configurations. The top row rendered by the real renderer R, and the bottom row is rendered by the neural renderer R.,
There are two main benefits modeling the environment. First, the reinforcement learning algo- rithm may have millions of steps, and it can be computationally expensive use the real environ- ment because sophisticated synthetic algorithms. save training time. Second. modeling the transition function of states using neural network makes the entire framework differentiable, and the loss can be backpropagated through the renderer net- work Rn.
we simplify the environment model, it can
4

---

Under review as a conference paper at ICLR 2020
In practice, model R, by building dataset consisting of paired data (ay, St, S(+1). To simplify the problem, we always start from blank canvas Su so that the R, only need to predict the stroke synthesize the blending function by making the stroke image from a single action. At run-time,
image transparent and adding it to the current canvas 8t- The paired dataset (a(s)) (()) collected by randomly sampling the a, and synthesising the corresponding 8;.
can be
We treat different paintbrushes of MyPaint (libmypaint contributors (2018) as separate models. worth noting that we use the environment model R2 for the training process, but use the real environment R, for the roll-out process.
5 CONSTRAINED PAINTING AGENT
After we train the natural media painting agent, we make the agent controllable to handle interactive scenarios and be used for generating stylized images.
To achieve this goal, one straightforward way is to constrain the agent with a modified action space. However, this method is difficult to generalize to different constraints. For each different constraint e in the same subspace C' in action space A, the policy needs to be retrained. For example, if we train a painting agent and constrain its stroke width 0.8, we still need to train it again when need an agent to output a stroke with width 0.9.
In the rest of this section, we propose constrained painting agent that, follow any constraint drawn from the subspace of action space A. First, we define the constraint representation, including the definition of the constraint, the encoding function, and the decoding function. Then introduce the training and roll-out schemes of the constrained agent.
5.1 CONSTRAINT REPRESENTATION
The constraint is the vector in the constraint space C, which the sub-space of the action space A. While fixing the vector the agent has to explore the subtraction of A and C' as A' by sampling a' from A' and concatenating with c:
a a= a' E A' = =A-
We can define C' by selecting and combining dimensions in A. For example, we can constrain the color of the stroke by defining C the color space (R, G, B). We can constrain the start position of the stroke by defining C as (Io,21). 21). We can constrain the stroke width of the stroke by defining C' as (r). Moreover, we can use the combination of the constraints to achieve complex effects.
As shown in Figure 3) the unconstrained agent takes the reference image and the current canvas observation,
(s", 8x) as observation while the constrained agent takes the reference (s" 88, G2) which has an additional constraint at step t. C is a vector drawn from the sub-space of the action space. To unify the observation representation for implementation, we encode the constraint vector as a bitmap to feed into the CNN of the policy network and the value network.
To encode the constraint C into observation, we upsample the constraint e; as matrix ef which has the same size of s* and st, to stack them and feed into our policy network and our value network. To decode the constraint of into a, we implement downsample module within the policy network. The policy network pi can be seen separate policy: ne and тa; Te outputs the downsampled constraint c and Ta outputs the constrained action a'. Then we concatenate a' and to form action
Formally, we have e =
=To((/ Te(c)), di = Ta("", 8t, (r,)), and a; = de + Cr.
5.2 TRAINING THE CONSTRAINED PAINTING AGENT
After we introduce the constraint a part of the observation of the painting agent, propose the corresponding training scheme. Because the constraint representation is designed for interactive purposes, need to either human experts another agent to train the constrained agent.
As shown in Figure 3 we use an unconstrained agent to generate constraints by cascading them together. For each step I, the unconstrained agent takes the reference image and the current canvas
5

---

Under review as a conference paper at ICLR 2020
(s", St) observations, and outputs the action in action space A. Then identify subspace in as constraint space and transfer a to C₁ followed by upsampling the c as c, as defined in section 4 After that, the constrained agent takes the additional constraint o, as an observation St, cc) and outputs action concatenated by cz and a'.
Canvas
Unconstrained Agent
Constrained Agent
Reference
R
Reference
Encoded Constraint
User defined constraint
Figure 3: Framework for Training (Roll-out) the Constrained Agent We train the constrained agent by cascading unconstrained agent. First, identify subspace of the action space and extract the constraint e from the action a. Then we upsample the constraint e and pass it on as an observation of the constrained agent. We use R2 renderer. For roll-out process (dashed line), we extract the constraint from either the user defined constraint or the action of the previous step. We use real renderer R to get the best visual quality.
5.3 ROLL-OUT CONSTRAINED PAINTING AGENT
For the roll-out of the constrained painting agent, we replace the unconstrained agent from training scheme in Figure 3 with the constraint extracted from either human command painting agent's 's output for each step t.
When the constraint is extracted from human command, the agent can be ordered to place paint stroke with a given position (mo, yo). color (R, G, B), and stroke width r. When the constraint extracted from painting agent's output, it can be seen that split the agent that explores in entire action space into two agents that explore separate action spaces An and A,, where A Ao + A1.
6 RESULT
6.1 NATURAL MEDIA PAINTING ENVIRONMENT MODEL
For training the natural media painting environment, treat different paintbrushes different models. We run 50,000 iterations for each model and record the la loss on the validation dataset as shown in Table 2
The learning curves of the training processes shown in Figure 4 (left).
Table 2: la loss of paintbrush models
Paintbrush
4 Loss
Paintbrush
l2 Loss
charcoal
2.12
10-3
impressionism
2.43 x 10-3
pencil
8.37 X 10
marker
5.01 x 10
bulk
9.16 X 10
dry brush
3.06 x 10-3 3
calligraphy
5.92
10-4
watercolor
1.16 x 10-4
6.2 UNCONSTRAINED PAINTING AGENT
We train the unconstrained painting agent using the fixed envionment model with various dataset. We use hand-written digits images (MNIST LeCun & Cortes (2010)), character images (Kanji Apel 2014)) face images (CelebA Liu et al. (2015), object images (ImageNet Deng et al. (2009) train painting agents. We run 10,000 episodes for each training task. Because of the difference among the dataset, we use 5 strokes to reproduce hand-written digits images, 20 strokes to reproduce character images, 100 strokes reproduce face and object images.

---

Under review as a conference paper at ICLR 2020
We measure the l2 loss between the reference images and reproduced images throughout the training process shown as Figure 4 4mmiddle.. The reproduced results shown as Figure and the corre- sponding l2 loss is shown Table 3]lff).
0.03
0.53
back
caligaphy
10-7
0.41
0.004
1001
0.83
2.00
0.22
1071
6.11
0.01
Toa TORTRO
345
Figure 4: l2 Loss over the Course of Training. We compute the l2 distance between the roll-out results and the reference images on validation dataset. (Left) is the course of training the neural renderer R, using various paintbrush. (Middle) the course of training the unconstrained agents us- ing varisous dataset. (Right) is the course of training the constrained agents using various constraints configurations.
Table 3: l2 training loss of training schemes
Training Scheme: Dataset
l Loss
Training Scheme: Constraint
l Loss
MNIST KanjiVG ImageNet CelebA
8.16 X 10
width (r)
1.25 x 10-2
4.02 10
start position (zo, y0)
1.43 X 10
7.06 X 10
color (R, G, B)
1.18 x 10-2
8.55 x 10-3
all above (To, Yo. r, R, G)
1.63 x 10-2
4
4
9
MNIST
KanjVVG
CelebA
ImageNet
Figure 5: We trained our natural media painting agents using MNIST, KanjiVG, CelebA, and Ima- geNet as reference images (left). We generate results (right) using R, for the training and validating process.
For the roll-out process, we employ a coarse-to- to-fine strategy to increase the resolution of the result shown as Figure 6 First, we roll out the agent with a low-resolution reference image s* and get the reproduction s* Then we divide 8* and s* into patches and feed the agent as initial observation. The roll-out results using various natural media are shown Figure
6.3 CONSTRAINED PAINTING AGENT
We train the constrained painting agents using the learning parameters as unconstrained painting agents. We compute the la distance between the reference images and reproduce results for the training process with various constraint configurations. To control variates, we use same neural environment R., (charcoal) and dataset (CelebA) for these experiments. We use the color (R, G, B), the stroke width r, the start position (zo, yo) and all of them (æo, yo. T, R, G. B) constraints.
The learning curves of these constraints are shown as Figure 4[right) and the 12 loss is shown as Table 3]ighht). We demonstrate the constrained painting agents as Figure 2 which uses pencil as paintbrush and incorporates coarse-to- toorse--t-fiiee fine strategy by dividing the reference images as 4x patches. To constrain the color of strokes, build color palette by clustering the colors of the reference image. For each action a, we constrain from a color randomly selected from the color palette. To constrain the stroke width, use constant stroke width for the roll-out process.
7

---

Under review as a conference paper at ICLR 2020
1x1 patch,
4x4 patches,
100 strokes
1600 strokes
Figure 6: Coarse-to-fine Roll-out We roll out the trained agent using coarse-to-fine strategy. We first roll out the model treating the reference image and canvas as one patch. Then we divide the reference image and updated canvas into patches and feed to the agent. In this figure, we roll 100 strokes in each patch.
charcoal
pencil
calligraphy
bulk
Reference
watercolor
dry brush
marker
impressionism
Figure 7: Exemplary Resuli Results generated by the painting agent using different paintbrushes. We demonstrate our natural media painting agent by rolling out agents trained with various paintbrushes from MyPaint.
reference
low contrast colors
high contrast colors
large stroke width
small stroke wichth
Figure 8: Roll-out Using Constrained Painting Agents We constrain the stroke width and the color of the agent to get the stylized results.
7 CONCLUSION AND FUTURE WORK
In this paper, we train natural media painting agents that can generate artistic paintings using various natural media, and collaborate with humans and other agents to get different visual effects. We build a model of natural media environment using deep CNN and train a natural media painting agent using model-based reinforcement learning. introduce controls the agents for interactive purposes, we propose constraint representation, framework for training a constrained painting agent, and various roll-out schemes to apply the agent. We demonstrate our algorithm by applying the trained model using various paintbrushes from MyPaint and constraints set up. The experimental results show that our algorithm can reproduce reference images in multiple artistic styles.
For future work, aim to extend the proposed algorithm by building unified model for differing paintbrush configuration. In addition, we will train a hicrarchical agent that uses constrained agent the low-level policy. We would like to apply our approach other reference images and use for interactive painting systems.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Ulrich Apel. Kanjivg. kanji SVG dataset, Creative Commons BY-SA, 3, 2014.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, SM Eslami, and Oriol Vinyals. Synthesizing programs for images using reinforeed adversarial learning. arXiv preprint arXiv:1804.011 00100.0111.. 18, 2018.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. A neural algorithm of artistic style. arXiv preprint arXiv: 06576, 2015,
lan Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
David Ha and Douglas Eck. A neural represennaatinn of sketch drawings. CoRR, abs/1704.03477, 2017a. Utthttp:: rxi wttp://rg//bbb///000040000
David Ha and Douglas Eck. arXiv: 2017b.
neural representation of sketch drawings. arXiv preprint
Aaron Hertzmann. Painterly rendering with curved brush strokes of multiple sizes. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques, pp. 453-460. ACM, 1998.
Aaron Hertzmann, Charles E Jacobs, Nuria Oliver, Brian Curless, and David H Salesin. Image analogies. Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp. 327-340. ACM, 2001.
He Huang, Philip S. Yu, and Changhu Wang. An introduction to image synthesis with genera- tive adversarial nets. CoRR, abs/1803.04466 2018. UKLLhttp.//xxx...orr/aa///
04469
Zhewei Huang, Wen Heng, and Shuchang Zhou Learning to paint with model-based deep rein- forcement learning. arXiv preprint 1rXiv:1903.04411, 2019.
Biao Jia, Jonathan Brandt, Radomir Mech, Byungmoon Kim, and Dinesh Manocha. Lpaintb: Learn- ing to paint from self-super ArXiv, 2019a.
Biao Jia, Chen Fang, Jonathan Brandt, Byungmoon Kim, and Dinesh Manocha. Paintbot: A rein- forcement learning approach for natural media painting. arXiv preprint arXiv: wrrXiv:1904.022. 1904.0221,, 2019b.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv:1710.10196 2017.
MP Pavan Kumar, Poomnmaa, HS Nagendraswamy, and Manjunath. A comprehensive survey on non non- -on-photorealistic rendering and benchmark developments for image abstraction and stylization. Iran Journal of Computer Science, pp. 1-35, 2019.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URLattp://yann.
lecun wwt//xdb/mm...eeeeeo
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Universal style transfer via feature transforms. In Advances in Neural Information Processing Systems, pp. 386- 396, 2017.
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Flow-grounded spatial-temporal video prediction from still images. In Proceedings of the European Conference Computer Vision (ECCV), pp. 100-615, 2018.
libmypaint contributors. libmypaint. http://gthhaa.commnaaaiiiiiinnn 2011 2018

---

Under review as a conference paper at ICLR 2020
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint MXXh:11000009711,, 2015.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. Scribbler: Controlling deep image synthesis with sketch and color. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, 2017.
Fan Tang, Weiming Dong, Yiping Meng, Xing Mei, Feiyue Huang, Xiaopeng Zhang, and Oliver Deussen. Animated construction chinese brush paintings. IEEE transactions on visualization and computer graphics, 24(12):011-33333 2018.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances In Neural Information Processing Systems, pp. 613-621, 2016.
Georges Winkenbach and David H Salesin. Rendering parametric surfaces in pen and ink. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pp- 469-476. ACM, 1996.
Ning Xie, Hirotaka Hachiya, and Masashi Sugiyama. Artist agent: A reinforcement learning ap- proach to automatic stroke generation in oriental ink painting. CoRR, abs/ 206.4634, 2012.
Ning Xie, Tingting Zhao, and Masashi Sugiyama. Personal style learning in sumi-e stroke-based rendering by inverse reinforcement learning. Information Processing Society of Japan, 2013.
Ning Xie, Tingting Zhao, Feng Tian, Xiao Hua Zhang, and M Sugiyam. Stroke-based stylization learning and rendering with inverse reinforcement learning. IJCAI, 2015.
Ningyuan Zheng, Yifan Jiang, and Dingjiang Huang. Strokenet: A neural painting environment. In International Conference on Learning Representations, 2019.
Tao Zhou, Chen Fang, Zhaowen Wang, Jimei Yang. Byungmoon Kim, Zhili Chen, Jonathan Brandt, and Demetri Terzopoulos. Learning doodle with deep q networks and demonstrated strokes. British Machine Vision Conference, 2018.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 2223-2232, 2017.
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
Reference
rounded
short grass
dry brush
watercolor
marker
pencil
watercolor
Figure 9: Roll-out results using Various Paintbrushes We roll out our natural media painting agents trained with various brushes MyPaint. To increase the resolutions of the generated images, we incorporate the coarse-to-fin to-fine strategy. We use 8 8 patches for first row and 4 x 4 for second row.
Figure 10: Reproduction of Starry Night using Charcoal We roll out natural media painting agent trained with charcoal brush in MyPaint to reproduce Van Gogh's 's starry night. We incorporate the coarse-to-fine strategy by dividing the reference image and canvas into 16 x 16 patches.
11

---

Under review as a conference paper at ICLR 2020
Figure 11: Reproduction of Starry Night using Watercolor We roll out our natural media painting agent trained with watercolor brush MyPaint to reproduce Van Gogh's 's starry night. We incorporate the coarse-to-fin tofe strategy by dividing the reference image and canvas into 16 x 16 patches.
12


Reviews:
[{"review_id": "Hyl4lLkRKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a reinforcement learning agent trained to interact with a painting environment in order to reproduce target images. The novel aspect of this work seems to be a version of the agent producing partial actions (i.e., portion of the action tuple is clamped to pre-specified values). This new agent receives the clamped components of the action as an additional conditioning input.\n\nPros:\n+ In some cases, images produced by the system look appealing.\n\nCons:\n- The writing of the manuscript could be significantly improved. I had a hard time understanding certain parts of the paper (e.g., what “constrained” means in the context of the present work). I got an impression that there was an effort to make things look more complex than they really are.\n- The proposed model lacks novelty – there seems to be only one non-trivial contribution and I’m not entirely sure how useful it is. The authors never compare their system against a simple baseline when one just overrides the actions of the agent.\n- In general, the evaluation section of the paper leaves a lot to be desired. The authors report some numbers (most of which are not even for the proposed model) but I’m struggling to make anything out of that information. Why should I care about them? What interesting conclusions can I draw from them? The paper never discusses this. On top of that, there are no baselines.\n\nNotes/questions:\n* At test time, the policy receives renders from a real environment. Could that create problems since it has only been trained on images synthesized by a neural surrogate?\n* Abstract: “on many benchmarks” -> “on several benchmarks”\n* Section 1: “we use a constraint representation along with …” – this sentence needs to be rewritten.\n* Section 2: Missing reference – Neural Painters (Nakano, 19). Considers a neural surrogate of the libmypaint environment.\n* Section 2, last paragraph:  “both methods can work on either a small action space or a small observation space” – how is the present model different? The action space is very similar (albeit continuous) to the existing approaches (i.e., relies on Bezier curves much like in (Ganin et al., 18)).\n* Section 2, last paragraph: The last sentence needs to be rewritten (“uncontrollable agent” looks a bit strange)\n* Section 3, first paragraph: “We highlight all …” -> “We describe all …”\n* Section 4.1, first paragraph: The first sentence needs to be rewritten (“the corresponding canvas by the given action”).\n* Section 4.1, first paragraph: “Unlike the previous …” – not true. (Ganin et al., 18) proposes to use this environment and (Nakano, 19) trains a neural surrogate for it.\n* Section 4.4: “WGAN loss (Huang et al., 19)” – this WGAN loss was introduced in (Ganin et al., 18).\n* Section 5: I feel like the paper could do a better job at explaining what “constraining” really means and justifying why it’s an interesting problem to solve. In my opinion, Eq. (3), for example, obscures rather than clarifies the notion of “constraints”.\n* Section 5, paragraph 2: “For each different” -> “For each”\n* Section 5.1, paragraph 4: “pi” -> “\\pi”\n* Figure 5: The caption almost overlaps with the text below.\n* Section 6.3, paragraph 2: “l2” -> “l_2”\n* Section 7, paragraph 2: “differing” -> “different”\n\nI feel like the authors should perform a major re-writing of the manuscript before it’s ready for publication. Moreover, I failed to see any significantly novel aspects of the proposed system (maybe due to the poor presentation) and therefore I wouldn’t recommend the paper for acceptance.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkehXzoP9H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an RL agent for generating a painting from a photograph by optimizing a sequence of brush strokes to match the target image.\n\nI believe the paper should be rejected because it does not have significant technical novelty for a first-tier conference, and the results do not show much aesthetic or technical advance. In terms of technical novelty, the paper seems to be applying a standard RL agent to an existing problem space, to optimize existing losses. The addition of constraints is technically very simple. The paper itself fails to articulate a compelling statement of novelty, for example, in discussing Huang 2019, the paper just says that a limitation of that method is that it uses OpenCV and that it doesn’t provide a system for control. Removing OpenCV is not a publishable contribution and shouldn’t even be mentioned; the control mechanism is not particularly novel.\n\nSome of the results do look nice, but it’s hard to say that the method has improved over the past 20 years of stroke-based rendering, e.g., many of the results look worse than those in Hertzmann 1998. The paper doesn’t offer any meaningful comparisons to the previous work, such as fair side-by-side comparisons on the same images, comparing computation times and aesthetics. The paper doesn’t articulate any aesthetic goals or state any meaningful standard by which the images might have improved over previous work. The control mechanism is not tested in any meaningful way (e.g., user studies), and the changes it makes to the results seem fairly trivial (e.g., contrast can be added to an image just as well via a post-process).\n\nI’m not sure what ICLR’s policy on citing previous unrefereed work is, but one paper that seems to have much more interesting results in a similar context is: https://arxiv.org/abs/1904.08410 . Reiichiro Nakano, Neural Painters: A learned differentiable constraint for generating brushstroke paintings.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "Byes_E3hYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present the results of training a natural media painting agent using reinforcement learning for different types of strokes. The agent seems to be capable of learning how to pain under different types of constraints and produce visually interesting images.\n\nComments:\n\n- Given that the authors give an implementation of the constrained RL agent as one of the key contributions of the paper, there is a glaring absence of mentioning related work on constrained reinforcement learning and reviewing the existing approaches in literature, in order to compare and contrast what the authors propose in this paper. This makes it hard for the readers to assess the novelty of the contribution.\n\n- Similarly, the authors should discuss in more detail the limitations on the types of constraints that are possible to easily express in this framework, given the simplicity of the constraints that are shown in the experiments\n\n- The readability of the paper seems like it could be improved. Apart from typos, there seem to be many long enumerations of approaches that other researchers have taken in this space, but for a reader it is not immediately obvious how these come together and relate to the work that is being presented.\n\n- With the above in mind, one thing that was conceptually unclear to me is that one of the main advantages of the proposed approach, according to the authors, when compared to some of the cited related work, is that this approach can generate intermediate representations and not just the final output with most resemblance to the reference picture. There is a mention of this opening up new artistic possibilities. Yet, this particular use case is not given central stage in evaluation. The authors should provide more examples of why this particular capability is relevant and how it leads to interesting outcomes.\n\n- The authors say: “we use 5 strokes to reproduce hand-written digits images, 20 strokes to reproduce character images, 100 strokes to reproduce face and object images” - which does seem reasonable, but the actual numbers (5, 20, 100) are not well motivated. After all, why not (10, 50, 200) or (5, 40, 100) or (5, 40, 200)? It would be good to show experimentally (for one of these) that the choice is justified. I’m not familiar with KanjiVG - but there do exist kanji characters with more than 50 strokes. I’m guessing they are not present in this dataset?\n\n- The authors should provide more details on the specifics of the model architecture and the hyperparameters that have been used / explored.\n\n- In the results/discussion of the paper, the authors should compare to prior work and highlight their novel contributions. Given that multiple papers out there that have had at first glance similar-looking results, it is hard to otherwise qualitatively judge whether what is proposed in this paper is better, without any form of side-by-side comparison.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper is withdrawn by authors.
Decision: Reject