Image style transfer via curved stroke rendering

目的 针对GANILLA、Paint Transformer、StrokeNet等已有的风格迁移算法存在生成图像笔触丢失、线条灵活度低以及训练时间长等问题，提出一种基于曲线笔触渲染的图像风格迁移算法。方法 首先按照自定义的超像素数量将图像前景分割为小区域的子图像，保留更多图像细节，背景分割为较大区域的子图像，再对分割后的每个子区域选取控制点，采用Bezier方程对控制点进行多尺度笔触生成，最后采用风格迁移算法将渲染后的图像与风格图像进行风格迁移。结果 与AST （arbitrary style transfer）方法相比，本文方法在欺骗率指标上提升了0.13，测试者欺骗率提升了0.13。与Paint Transformer等基于笔触渲染的算法对比，本文能够在纹理丰富的前景区域生成细粒度笔触，在背景区域生成粗粒度笔触，保存更多的图像细节。结论 与GANILLA、AdaIN （adaptive instance normalization）等风格迁移算法相比，本文采用图像分割算法取点生成笔触参数，无需训练，不仅提高了算法效率，而且生成的多风格图像保留风格化图像的笔触绘制痕迹，图像色彩鲜明。;Objective The goal of the image style transfer algorithm is to render the content of one image with the style of another image. Image style transfer methods can be divided into traditional and neural style transfer methods. Traditional style transfer methods can be broadly classified to stroke-based rendering(SBR)and image analogy(IA). SBR simulates human drawings with different sizes of strokes. Meanwhile, the main idea of IA is as follows:given a pair of images A (unprocessed source image)and A'(processed image)and the unprocessed image B, the processed image B' is obtained by processing B in the same way as A to A'. Meanwhile, neural style transfer methods can be categorized into slow image reconstruction methods based on online image optimization and fast image reconstruction methods based on offline model optimization. Slow image reconstruction methods optimize the image in the pixel space and minimize the objective function via gradient descent. Using a random noise as the starting image, the pixel values of the noise images are iteratively changed to obtain a target result image. Given that each reconstruction result requires many iterative optimizations in the pixel space, this approach consumes much time and computational resources and requires a high time overhead. In order to speed up this process, fast image reconstruction methods are proposed to train the network in advance in a data-driven manner using a large amount of data. Given an input, the trained network only needs one forward transmission to output a style transfer image. In recent years, the seminal works on style transfer have focused on building a neural network that can effectively extract the content and style features of an image and then combine these features to generate highly realistic images. However, building a model for each style is inefficient and requires much labor and time resources. One example of this model is the neural style transfer(NST)algorithm, which aims at transferring the texture of the style image to the content image and optimizing the noise at the pixel level step by step. However, hand-painted paintings comprise different strokes that are made using different brush sizes and textures. Compared with human paintings, the NST algorithm only generates photo-realistic imageries and ignores paint strokes or stipples. Given that the existing style transfer algorithms, such as Ganilla and Paint Transformer, suffer from loss of brush strokes and poor stroke flexibility, we propose a novel style transfer algorithm to quickly recreate the content of one image with curved strokes and then transfer another style to the rerendered image. The images generated using our method resemble those made by humans. Method First, we segment the content image into subregions with different scales via content mask according to the customized number of super pixels. Given that we do not pay attention to the background, we segment the image background into small subregions. To preserve large amounts of details, we save the image foreground as much as possible via segmentation into small subregions. The segmentations for the image foreground are twice greater than those for the image background. For each subregion, four control points are selected in the convex hull of a subregion, and then the Bezier equation is used to generate thick strokes in the background and thin strokes in the foreground. The image rendered with strokes is then stylized with the style image by using the style transfer algorithm to generate a stylized image that retains the stroke traces. Result Compared with the arbitrary style transfer(AST)and Kotovenko' s method, the deception rate of the proposed method is increased by 0. 13 and 0. 04, respectively, while its human deception rate is increased by 0. 13 and 0. 01. Compared with Paint Transformer and other stroke-based rendering algorithms, our proposed method can generate thin strokes in the texture-rich foreground region and thick strokes in the background, thus preserving large amounts of image details. Conclusion Unlike whitening and coloring transforms(WCT), AdaIN, and other style transfer algorithms, the proposed method uses an image segmentation algorithm to generate stroke parameters without training, thus improving efficiency and generating multi-style images that preserve the stroke drawing traces of stylized images with vivid colors.