Training a Constrained Natural Media Painting Agent using Reinforcement Learning 

1 INTRODUCTION . Throughout human history , painting has been an essential element of artistic creation . There are many diverse and complex artistic domains with various styles such as watercolor , oil painting , sketching , and so on . As image processing and computer graphics have advanced , there has been a considerable effort to simulate these styles using non-photorealistic rendering ( NPR ) techniques ( Kumar et al . ( 2019 ) ) . Hertzmann ( 1998 ) ; Winkenbach & Salesin ( 1996 ) generate compelling results using stroke-based rendering . However , most prior methods in NPR are engineered for a specific application or task , and can not easily adapt to new styles or medium . Recent developments in machine learning have resulted in significant advancements in computer vision and computer graphics , including computer-based painting systems . Many visual generative methods based on generative adversarial networks ( Goodfellow et al . ( 2014 ) ) as Zhu et al . ( 2017 ) ; Zhou et al . ( 2018 ) ; Huang et al . ( 2018 ) ; Karras et al . ( 2017 ) ; Sangkloy et al . ( 2017 ) have demonstrated promising results . Many of these machine learning methods have also been applied to stroke-based rendering tasks , including modeling the brush ( Xie et al . ( 2012 ) ; Zheng et al . ( 2019 ) ) , generating brushstroke paintings in an artist ’ s style ( Xie et al . ( 2015 ) ) , reconstructing drawings for specific paintings styles ( Tang et al . ( 2018 ) ) , and constructing stroke-based drawings ( Ha & Eck ( 2017a ) ; Zhou et al . ( 2018 ) ; Huang et al . ( 2019 ) ; Jia et al . ( 2019a ) ) . In this paper , we focus on a more general and challenging problem of training a natural media painting agent for interactive applications . Given a reference image , our goal is to develop a stroke-based rendering approach that can imitate the human drawing or strokes used in generating the image . A key challenge is to develop a method that can learn from scratch without any supervision . In this regard , we present a technique that can handle all inputs and train an agent to manipulate natural painting media such as charcoal , pencil , watercolor , and so on . We build a model-based natural media environment using deep CNN and train a natural media painting agent using model-based reinforcement learning . In order to introduce controls to the agents for interactive applications , we use a constraint representation along with a different framework for training and use the constrained painting agent . These constraints enable the agent to interact with a human or other agents and generate various styles without retraining the model . The novel contributions of our work include : • A method to train an agent that produces a stream of actions subject to constraint for each action . These constraints can include restricting the start location , stroke width , color , and other stroke parameters . • A method to roll out constrained agents so the user can produce new stylistic effects interactively or automatically , as the agent is painting by modulating the action stream . • By incorporate coarse-to-fine strategy , our painting agents can generate high-resolution stylized images using various constraints and paintbrush configurations . We evaluate our algorithm on different paintbrush configurations and datasets to highlights its benefits over prior reinforcement learning based methods . We also employ differing constraint settings to validate our constrained agents and produce new stylistic effects with a single trained model . 2 RELATED WORK . In this paper , we focus on the stroke-based rendering problem , which renders the reference image with brush strokes . In contrast to image analogy approaches ( Hertzmann et al . ( 2001 ) ; Gatys et al . ( 2015 ) ; Vondrick et al . ( 2016 ) ; Zhu et al . ( 2017 ) ; Karras et al . ( 2017 ) ; Sangkloy et al . ( 2017 ) ; Li et al . ( 2017 ; 2018 ) ) , stroke-based approaches can generate intermediate painting states for interactive purposes . They can also be deployed in various painting environment to generate different artistic effects . One category of approaches for stroke-based rendering uses a heuristics-based method to guide the agent while painting . Notable examples include the method in Winkenbach & Salesin ( 1996 ) , which can produce stylized illustrations , and the method in Hertzmann ( 1998 ) , which can reproduce a colorful painting in different styles . However , it is difficult to extend these methods to different styles of painting because of the hand-engineered features . Another category of approaches uses machine learning techniques to learn the policy . Compared with predefined policies , these methods use machine learning techniques to learn the painting policy , which enables the agent to generate a more natural result . Ha & Eck ( 2017b ) train an RNN to learn the latent space of the sketch data and generate the paintings in this latent space , which requires the paired dataset for training . Other approaches use deep reinforcement learning to learn the policy without supervised data . There have been a few attempts to tackle related problems in this domain . Xie et al . ( 2012 ; 2015 ; 2013 ) propose a series of works to simulate strokes using reinforcement learning and inverse reinforcement learning . These prior approaches learn a policy either from reward functions or expert demonstrations . Comparing with these approaches , we use a more general setup which does not rely on rewards engineering . Zhou et al . ( 2018 ) is based on the Deep Q network . Ganin et al . ( 2018 ) trains the discriminator and the reinforcement learning framework at the same time . However , both methods can work on either a small action space or a small observation space . Jia et al . ( 2019b ; a ) use proximal policy optimization with curriculum learning and self-supervision to gradually increase the sampled action space and the frequency of positive rewards . Zheng et al . ( 2019 ) train a differentiable painting environment model , which helps learn the painting policy with gradient-based methods . Huang et al . ( 2019 ) train a model-based RL using differentiable environment and DDPG ( Lillicrap et al . ( 2015 ) ) . This approach is limited by the OpenCV based renderer and the uncontrollable agent . 3 OVERVIEW . An overview of our approach is given in Fig . 1 This includes the training phase and extracting the underlying neural network as the painting agent is used for the roll-out process . We highlight all the symbols used in the paper in Table 1 . In section 4 , we present the details of natural media painting agent with a environment model for the agent to interact . Our approach is based on the training scheme described in Huang et al . ( 2019 ) and uses Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al . ( 2015 ) ) to train the modelbased painting agent . For the roll-out algorithm , we apply the painting agent directly to the real environment Rr as shown in Figure 1 . In section 5 , we present our representation of the underlying constraints and the underlying techniques to encode and decode these constraints using reinforcement learning . We use the unconstrained agent to identify the constraints , and encode them as observations to train the constrained painting agent , as shown in Figure 3 . In order roll out the constrained agent , we replace the unconstrained agent with user defined constraints or the action of previous step . 4 NATURAL MEDIA PAINTING AGENT . In this section , we present our algorithm for training a natural media environment model and training a painting agent to interact with this environment model . 4.1 RENDERER . The renderer of our painting agent generates the corresponding canvas by the given action , which implements the blending functions and other synthetic programs . Unlike the previous learning-based painting approaches Jia et al . ( 2019a ) ; Huang et al . ( 2019 ) ; Zheng et al . ( 2019 ) , we use a natural media painting renderer MyPaint ( libmypaint contributors ( 2018 ) ) for our experiment . Compared with self-defined painting simulators , it provides rich visual effects and sophisticated blending functions . Ganin et al . ( 2018 ) use the same environment but have not explored the various configurations of the paintbrush . Using a pre-existing environment setup helps our approach concentrate on the learning algorithm to make it easy to generalize . 4.2 ACTION REPRESENTATION . The action of our painting agent consists of the configurations it use to interact with the environment . For the action space , we follow the stroke definition of Ganin et al . ( 2018 ) using a quadratic Bezier curve ( QBC ) of three points ( ( x0 , y0 ) , ( x1 , y1 ) , ( x2 , y2 ) ) . Each stroke is a 3-point curve . The pressure which affects the blending function is linear interpolated by the value of the start position ( x0 , y0 ) and the end position ( x2 , y2 ) . We use one color ( R , G , B ) for each stroke , and the transparency of the pixels within the stroke is affected by the pressure ( p0 , p1 ) . We use one brush size r for each stroke , and the actual stroke width within the stroke can also be affected by the pressure ( p0 , p1 ) . Formally , we represent the action at as : at = ( x0 , y0 , x1 , y1 , x2 , y2 , p0 , p1 , r , R , G , B ) ( 1 ) In practice , at ∈ R12 . Each value is normalized to [ 0 , 1 ] . The action is in a continuous space , which makes it possible to control the agent precisely . 4.3 OBSERVATION REPRESENTATION . The observation of our painting agent consists of information to make decisions . In our setup , the observation is represented by the reference image s∗ , the current canvas st at step t , and the constraint ct . Formally , ot = ( s∗ , st ) . For the constrained agent described in section 5 , ot = ( s∗ , st , ct ) . ct is a vector drawn from the sub-space of the action space . To unify the observation representation for implementation , we upsample the constraint vector as a bitmap with size of st to feed into the CNN of the policy network and the value network . 4.4 REWARD . Reward is a metric that enables our painting agent to measure effectiveness of the action . Aside from the reinforcement learning algorithm , the definition of the reward can be seen as a guidance for the policy π . For our problem , the goal of the painting agent is to reproduce the reference image s∗ . First , we define the loss function L between the current canvas st at step t and the reference image s∗ . In practice , we use l2 loss ( Jia et al . ( 2019a ) ) and WGAN loss ( Huang et al . ( 2019 ) ) for implementation . Then , we use the difference of the losses of two continuous steps as the reward rt . We normalize rt using Eq.2 , such that rt ∈ ( −∞ , 1 ] . rt = L ( st−1 , s ∗ ) − L ( st , s∗ ) L ( s0 , s∗ ) , ( 2 ) where L is a loss function defined as l2 or WGAN . For our reinforcement learning setting , the objective is to maximize the discounted accumulative rewards qt = ∑tmax t=1 rtγ t , where the discount factor γ ∈ [ 0 , 1 ]

Summary: This paper presents a reinforcement learning agent trained to interact with a painting environment in order to reproduce target images. The novel aspect of this work seems to be a version of the agent producing partial actions (i.e., portion of the action tuple is clamped to pre-specified values). This new agent receives the clamped components of the action as an additional conditioning input.