Lmser-pix2seq: Learning Stable Sketch Representations For Sketch Healing
Authors: Tengjie Li
Sicong Zang
Shikui Tu
Lei Xu
Venue: Submitted to ICLR 2023 (2023)
Keywords: sketch healing
Lmser
stable representations
bi-directional connections

Abstract: Sketch healing aims to recreate a complete sketch from the corrupted one. The sparse and abstract nature of the sketch makes it challenging due to the difficulty in learning. The features extracted from the corrupted sketch may be inconsistent with the ones from the corresponding full sketch. In this paper, we present Lmser-pix2seq to learn stable sketch representations against the missing information by employing a Least mean square error reconstruction (Lmser) block, which falls into encoder-decoder paradigm. Taking as input a corrupted sketch, the Lmser encoder computes the embeddings of structural patterns of the input, while the decoder reconstructs the complete sketch from the embeddings. We build bi-directional skip connections between the encoder and the decoder in our Lmser block. The feedback connections enable recurrent paths to receive more information about the reconstructed sketch produced by the decoder, which helps the encoder extract stable sketch features. The features captured by the Lmser block are eventually fed into a recurrent neural network decoder to recreate the sketches. Experimental results show that our Lmser-pix2seq outperforms the state-of-the-art methods in sketch healing, especially when the sketches are heavily masked or corrupted.

Under review as a conference paper at ICLR 2023
LMSER-PIX2SEQ: LEARNING STABLE SKETCH REPRE- SENTATIONS FOR SKETCH HEALING
Anonymous authors Paper under double-bling review
ABSTRACT
Sketch healing aims to recreate a complete sketch from the compupted one. The sparse and abstract nature of the sketch makes challenging. The features ex- tracted from the corrupted sketch may be inconsistent with the ones from the corresponding full sketch. In this paper, we present Lmser-pix2seq to learn sta- ble sketch representations against the missing information by employing a Least mean square error reconstruction (Lmser) block, which falls into encoder-decoder paradigm. Taking as input corrupted sketch, the Lmser encoder computes the embeddings of structural patterns the imput, while the decoder reconstructs the complete sketch from the embeddings. We build bi-directional skip connections between the encoder and the decoder in our Lmser block. The feedback connec- tions enable recurrent paths to receive more information about the reconstructed sketch produced by the decoder, which helps the encoder extract stable sketch fea- tures. The features captured by the Lmser block are eventually fed into a recurrent neural network decoder to recreate the sketches. Experimental results show that our Lmser-pix2seq outperforms the state-of-the-ar methods in sketch healing, es- pecially when the sketches are heavily masked or corrupted.
INTRODUCTION
Humans are able to complete things that are missing in life through their imagination, such com- pleting blanks, novel sequels and image repairs. Sketch healing task (Su (020) is one of the related works. Sketch healing is synthesise complete sketch that best resembles the partial input (Su all 2020 QI et al. ((2Z).. Different from the image inpainting task (Pathak et all, 2016), where photos have rich texture information, freehand sketches are highly abstract and sparse, mak- ing sketch healing quite challenging.
The way get a corrupted sketch, proposed by Sn et (2022)) is crop several local visual patches from raster sketch image and drop some of them. This approach results in corrupted sketch raster image and some remaining visual patches. Conventional sketch generation models (Chen al 2012 Zang et all 202T) that take images as input can be used for sketch healing. How- ever, these models designed for sketch synthesis are not comparable to SketchHealer- 1.0 (Su ctal 2020, which was specifically designed for sketch healing. SketchHealer-LO. constructs graphical representation of the sketch by treating patches as nodes and connecting edges based on the nodes' temporal proximity, i.e., the drawing order. The graphic sketch representation realizes the infor- mation interaction between different patches in the same sketch, so as to achieve a better effect of healing. Based on SketchHealer-I.0 SketchHealer-2.0 (Qi all, mI2R) considered the relationship between the local reconstruction and the global semantic preservation. SketchHealer- 2.0 requires the involvement of a pre-trained model to calculate the semantic similarity between the recreated sketch and the full sketch. SketchHealer- 1.0 (Su et all 2020) and SketchHealer-2.0 (QI et all 2022) build graphs that depend on drawing order, but this information not always available. To over- come this difficulty, Sketchl.tticc Lattice (Qi et all, 2021) proposes novel lattice representation and takes image as input. However, during the data processing phase, the lattice approach causes some of the information in the raster sketch image to be lost, thus limiting SketchLattic''s performance.
Different from the otate-of-the-art graph-structure models, which pass information between nodes to fill in the gaps, we expect that the network to take full advantage of the information in the raster sketch images and learn stable sketch representations in the absence of temporal information. Sta- ble representations mean that the model extracts the features of the corrupted sketch as consistent

---

Under review as a conference paper at ICLR 2023
as possible to the ones of the full sketch. Theoretically, this consistency allows different corrupted sketches obtained by masking from a full sketch to be recreated similarly. Conversely, when the extracted features are unstable (lack of consistency), the healed sketch fails to maintain semanties, and worse, its category changes.
To learn stable sketch representations, we expect the feature maps from different layers in the net- work to be fully fused, which helps to extract significant and stable features from the corrupted sketches. Least mean square error reconstruction (Lmser) (Xui, 199I; (993) enables this purpose. Lmser was a development of autoencoder (AE) (Bourlard & Kamp, (988) by folding and merging the symmetrical encoder and decoder together. Such folding equivalent to adding bi-directional skip connections between the encoder and the decoder (Xu, 2019). The effectiveness of Lmser demonstrated in image inpainting (Huang et all 20205), super-resolution (Lier all, 2019), and semantic segmentation (Guo 20T9; atl (021). However, these studies focus on image- related applications with rich texture information, rather than sparse and abstract sketches. We present Liser-pix2seq to learn stable sketch representations against the missing information by employing a Lmser block, which falls into encoder- decoder paradigm. Taking as input a corrupted sketch, the Lmser encoder computes the embeddings of structural patterns of the input, while the decoder reconstructs the complete sketch from the embeddings. We build bi-directional skip con- nections between the encoder and the decoder in our Lmser block. The feedback connections enable recurrent paths to receive more information about the reconstructed sketch produced by the decoder, which helps the encoder extract stable sketch features. The features captured by the Lmser block are eventually fed into a Recurrent Neural Network (RNN) decoder to recreate the sketches.
In summary, our contribution is that we propose Lmser-pix2seq to learn stable sketch representa- tions for sketch healing. The bi-directional skip connections in our Lmser blocks allow the feature maps from the encoder and decoder to be sufficiently fused to facilitate the extraction of sketch fea- tures. Experimental results show that our Lmser-pix2seq outperforms the state-of-the-art methods, especially when the sketches are heavily masked or corrupted.
2 RELATED WORK
Sketch Generation. Research related to sketch generation with deep learning methods (Ha & Fck, 2018: Zhou et all 2018: Das et all 2021: Gi et all (2021) has been developing rapidly in recent years. An interesting work on sketch generation is that the neural network imitates humans to draw the vector sketch stroke by stroke. sketch-rnn (Ha & EcR, 2018) is an RNN-RNN architecture generation model based on the Variational Autoencoder (VAE) (Kingma & Welling, 203)), which enables the conditional and the unconditional single category sketch generation. Later, the proposed sketch-pix2seq (Chen all 2017) with the convolutional neural network (CNN) encoder solves the multi-category generation problem and finds that latent code with the normal distribution constraint removed has better reconstruction results. Inspired by the above two models, (Song et al,, 2018) fuses photo texture information with temporal information by shortcut cycle consistency. To fur- ther improve the controllability of the generation, RPCL-pix2seq (Zang et all 2021) assumes that the latent space follows Gaussian mixture model (GMM), and the number of Gaussians is de- termined by automatic selection of the model. sketch-rnn and RPCL-pix2seq are more similar to model. There are some large pre-trained models, c.g. Sketch-Bert (Lin et ali, 2020) and Sketch- former (Ribeiro et all (02()), for not only the sketch generation, but also for other downstream tasks. Sketch healing is similar to the combination of vector sketch generation and image inpainting.
Sketch Healing. SketchHealer-11 1.0 (Su ei all 2020) clarified the definition of the sketch healing and proposed novel graph representation method. SketchHealer-2.0 (Qi all, 2022) rasterizes the gen- erated sequence and calculates its semantic perceptual loss from the corresponding full sketch. The other model that represents sketch as graph is the SketchLattice (Qi et all, 2020). SketchLattice is a lightweight network that can construct graphs without relying on the drawing order. SketchLattice lattices the image, then treats the intersection of the lattice with the pixels of the sketch stroke as the nodes, and constructs the edges between the nodes by Euclidean distance. In contrast to the sketch generation task, the sketch healing task requires the model to extract accurate and effective features when the sketch is masked.
Skip Connection. Forward connections refer to the direct transmission of information from the shallow layers to the deep layers via short-cireuit path. Studies have shown that forward connec- tions can alleviate the gradient vanishing problem (He et all, 2016) and promote multi-scale feature fusion (Ronneberger et all 2015). This technique widely applied in the field of computer vision,
2

---

Under review as a conference paper at ICLR 2023
Lmser
Awamge Reoling
Conv2d b2
DeComm2d 2639911222
Conv2d bi3 2226
DeCom22 5x3y925622
Lmssr Binck
RNA Decoder
Conv 1
I1
Conv2d b33664222
Corruptedd Sketch X
Convdd 5x 5xfg832/2
Dec 24 51588322
Canvd
Rasterize
Full Sketch
T kops
(a)
(b)
Figure 1: (a) Architecture of Lmsser block. The convolution and deconvolution configurations are shown as h w@d/s, where h, w, d and represent height, width, depth and stride, respectively. All the layers are followed by an instance norm layer and a relu activation function, except for the bottom-most deconvolution layer, which employs tanh activation funetion. (b) overview of Lmser- pixer... The latent vector y output at the top of the Lmser block is eventually fed to an RNN decoder to recreate the vector sketch S.
e.g., image recognition (He et 2016: Huang et all 2017: Dosovitskiy et all 2020) and image 2015: Milletan all 2016: Zhou at 20188 Huang atl 20203). Feedback connections, as opposed to forward connections, have received attention recently. The implementation of feedback connection usually requires the assistance of recurrent mechanism (Xiang et ar, 2020; Huang et arg 2020B). Studies on semantic segmentation of medical images have shown that feedback connections can acquire finer details to effectively improve the accuracy of segmentation (Xiang et all, 2020: Cao et all 202I).. In addition, block reuse reduces the number of parameters in the network.
semantic segmentation (Ronneberger
3
METHODOL.OGY
3.1 OVERVIEW OF LMSER-PIX2SEQ
The proposed Amser-pix2sen a model consisting an Lmser block encoder and an RNN decoder, where the Lmser block is also in encoder-decoder architecture. The bi-directional skip connections our Lmser block allow the feature maps to fully fused extract stable features from the sketch. Fig. II(b) shows the pipeline of our Lmser-pix2seq. A corrupted sketch X is fed into the Lmser block to obtain the reconstructed raster image X and the latent vector y. respectively. We project into two vectors μl and a and then get the final latent code m by the reparametrisation trick (Kingma & Welling 201)), which is fed into Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuben. 1997) decoder output the recreated sequence S. Specifically. the LSTT predicts the pen and produces parameters to form a GMM to estimate the offset distance from previous point. Our LSTM decoder with 512 nodes is consistent with the SketchHealer-1. (Su 2020). The LSTM modeling method is referenced from the sketch-rnn (Ha & Eck. 2018), from which readers can access more details.
In our method, a sketch has sequence Š( $2, .... Sn) and raster image X two different modal representations. A sequential representation of the sketch is composed which is a vector containing five elements: (Ar, Ay, Pl, P1,P2,P0 Pa). Ar and Ay are the offset distance from the previous point
series of points, each of
x and y direction. (p1. P2.P3) indicates the current stroke state, where 0, (0, 1. O), (0, 0, 1) represent the three stroke states of touch, lift, and the end of sketch, respectively. X is rasterized by the sequence S. The corrupted sketch X is obtained by masking part of the information in the full sketch X.

---

Under review as a conference paper at ICLR 2023
3.2 LMSER BLOCK ARCHITECTURE
As mentioned before, we expect a network to recover incomplete sketch raster images while extract- ing features. We take convolutional autoencoder (CAE) as the backbone network implement our Lmser block. The encoder and decoder of the CAE consist of several convolution and decon- volution layers, respectively. as shown in Fig. I[.). The encoder and decoder layers of the CAE are fully symmetrical. We add bi-directional skip connections to the corresponding layers between the encoder and the decoder. We connect the decoder and the encoder to form a loop, as illustrated in Fig. Mīa). that allows the feedback skip connections to work (Huang et al. 20206). With the recurrent mechanism, we calculate the feature fusion as follows:
L 0,0 0,000<<44
2 te ( +1- (1)-))2(- (1-I)
t> 0,0 < L,
2 fz
12 0,1=1,
(1)
2) (2 2t ag a1 (xf+1) a1= 12 0,000<<+
12 0,1=0, 0,
where z/ and z/d respectively denote the 1-th update of the 1-th layer's feature maps from the encoder and the decoder, ('()) and g') separately denote the 1-th layer network-related calculations for the of encoder and the decoder, is hyper-parameter, which we set to 0.5 in the experiments. We use the bottom-mot output z of the decoder as the input to the encoder to form the loop, i.e.,
(I+1)e =
20
The specific workflow of the Lmser block is as follows: when a sketch X is fed to the Lmser block, the neurons of the encoder activated layer by layer. Then, the embedding #1, obtained from the top-most layer of the encoder are sent to the decoder. The neurons in each layer of the decoder are initialized by receiving both the information transmitted by the forward connection and the signals from the previous layer of the decoder, e.g. >²1h ag) (22r) (1 a)²{* The decoder output, =0
.d is used as the input zlle to the encoder in the next loop. Thereafter, the updates of the neurons in the encoder are also influenced by the information delivered by the feedback connections, e.g- z/ = (= + (1 a)2lá. The neurons in the Lmser are dynamic updated layer by layer through a large loop repeatedly, see Fig. 2 of CLmser (Huang et al, 20200) for more details.
After 7' loops, the top-most and bottom-most layers of the Lmser block output the feature and the reconstructed sketch X, respectively. Theoretically. the dynamic process above can eventually reach an equilibrium state (Xu, (193). In practice, the choice of I is a trade-off between performance and consumption. More iterations allow the feature maps to be fully fused and facilitate the output of stable features, but also require the involvement of more computational resources. In this paper, we set T to 2 by default. It is worth noting that when T = 0, Lmser will degenerate into a U-Net-like network architecture.
3.3 TRAINING LMSER- PIX2SEQ
minerrppx2ses has multi-modal output property whose main purpose is to make the recreated vector sketch S as high-quality as possible while maintaining global semantic with the original full sketch S. To obtain better reconstruction in multi-class sketch, we remove the Kullback-Leibler (KL) Divergence Loss term in VAE as existing work (Chen al 2017: Su all 2020 QI et al,, 2021 2022). The loss function consists of two parts: the raster image reconstruction loss &CNN and the sequence reconstruction loss LNNN- As commonly image generative tasks, LCN n adopts the 12 loss. Following sketch-rnn rnn (Ha & EcK, 2018), the sequence modal's reconstruction loss RRNN is to minimize the negative log- likelikeldhood of the generated probability distribution (Qi et all 2021).
To sum up, our objective is to minimize
L total =w.bONN W.C.RN
1
2 - X
m)), m)
where X and X denote the full sketch and the complete sketch reconstructed by Lmser block, respectively. LCNN and L NNN weighted by hyper-parameters w, and w,, correspondingly.
4

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENT
4.1 PREPARATION
Dataset. Our experiments are conducted on two datasets, a 1 17-category dataset (DS1) from
SketchHealer-1.0 1.0 (Su all (020) and 5-category dataset 2 (DS2) from RPCL--pi22seq (Zang
aLa 202I). DSI and DS2 are both from the large-scale dataset Quickdraw (Ha & Eck, 2018)
Each category contains 70000 sketches for training and 2500 sketches for testing.
Our coruptiion method is generally consistent with SketchHealer-11 1.0 (Su 2020). The sketch is drawn with 640 640 canvas, and M patches of 128 x 128 are taken from the canvas according to the order of the stroke points, then some of the patches are removed with a probability Pmosk. In the end, we resize the corrupted canvas into a 128 x 128 image as the corrupted sketch. SketchHealer- 1.0's mask method (Su et all, ((20)) has an information leakage problem. There are two patches A and B with overlapping parts, A is cropped before B, and B is selected to be removed, at this time, A still contains some information from the removed B. Our solution to the information Ieakage problem in SketchHealer-I1 1.0 (Su et all, (020) is to first mask all the information on the canvas corresponding to the location of the patches that need to be removed, and then crop the individual patches.
Evaluation Metries. Rec and Rel (Zang et all 2024T) are evaluated recreated sketch metrics. Rec indicates whether the recreated sketch S and its corresponding sketch S belong the same category, e.g. top 1 recognition accuracy. We pre-train two sketch net classifiers (Yu 2015) for computing in two datasets, respectively. Ret a measure of the semantic similarity between the recreated sketch Š and its corresponding full sketch S. Specifically, for different models, full sketch S and recreated sketch S are rasterized and fed to their own encoder to obtain vectors jü and j.. We use f to retrieve its corresponding vector ft, and Ret is the success rate of retrieval. We use Ret@k to represent top- retrieval accuracy. Our Ret is completely different from SketchHealer- 1.0 (Su et all 2020) SketchHealer-2.0 (Qi et all, 2022) and SketchLattice (Q et all 202I), whose retrieval is category- retrieval and not an instance-level retrieval. Our Ret is more responsive to the semantic consistency of the recreated sketch and the full sketch.
Baseline. We compare our Lmser-pic pisee with five baseline models in the sketch healing task. These models include sketch-pix2seq (Chen et all 2017) and RPCL-pix2seq (Zang all 2021), which are proposed for sketch generation. Additional models are SketchHealer- (Su et all, 202(),
SketchHealer-2.0 etl 2022), and SketchLattice (Qiet all M02T), which are designed for sketch healing. Among these models, only RPCL-piX2seq has constraints on the latent space.
We train SketchLattice (Qi et all (02T) and SketchHealer-1 1.0 (Su et all (020) using the official open source code. Mcanwhile, we re-implement RPCL- by pytorch (Paszke et all 2017) The code of SketchHealer-2.0 (Qi et all 2022) is not open source yet. while the original paper lacks the specific implementation of gradient back propapppaaationn after rasterization. Therefore, we do our best to reproduce SketchHealer-2.0. We first pre-train multi-category sketch-rnn & EGK, 2UI) networks for each datasets and freeze the networks' parameters. When training SketchHealer-2.00 we choose the Gaussian component and pen state with the highest probability of the LSTM output to generate a five-tuple vector sequence Å to ensure that the gradients can be mack-propagated. Eventually, $ and S are simultaneously fed into the pre-trained sketch-rnn's encoder to get the corresponding latent code, which are used to calculate the perceptual loss. Since the masking process is performed on the canvas, we do not include networks like sketch-rnn, which use sequences as input, as the baseline models.
Implement Details. The Adam optimizer (Kingma & Ba, 2014) is applied to our Lmser-pix2seq with parameters 3₁ 0.9, 52g 0.999, 10 8, and the learning rate of the network starts from 10-3 with a decay rate of 0.999 for every iteration. We randomly mask Pmask = 10% of the patches to obtain the corrupted sketch for model training, and different proportions for model testing. The weights of the loss function are w, 0.5 and wi = 1. The dimension of the latent code m is N₂ = 128. All models is trained on a single NVIDIA RTX 2080Ti GPU with 150000 iterations.
airplane. angel, alarm clock. apple. butterfly, belt, bus, cake, cat, clock, eye, fish, pig. sheep, spider, umbrella, The Great Wall of China. These categories are common in life and the instances the categories are globally similar in appearance. bbee, bus, flower, giraffe, pig. These classes have multi-style characteristics and are more challenging for sketch healing.
5

---

Under review as a conference paper at ICLR 2023
4.2 RESULTS
Quantitative Results. Table II reports the retrieval performance on DS1. The proposed Lmser- pix2seq learns stable representations which lead to outstanding performance Ret. When the sketch is unmasked, the success rate Lmser-pix2seq retrieval performs significantly superior to other methods. The top-1 Ret of our method is 22.91% higher than SketchHealer-1.0 (Su ala 2020). The advantage the Lmser-pix2seq becomes more apparent when the proportion of sketches masked gradually increases. When 550 the top1 Ret of Lmser-pix2seg improved by 33.42% compared with SketchHealer-Ie well, see Table Ø The Rec metrics of imser-pix2sse of-the-art methods SketchHealer-1. 1.0 (Su all 2020) and SketchHealer-2.01 (Qi et all 2022). The above results show that our model satisfies the need for both high-quality and semantic preserva- tion. Observe the performance of other models. RPCL-pix2seq (Zang et al, 20241,, as a latent space constrained model, focuses on the controllability of sketch generation rather than sketch healing, which makes it more sensitive to masking rates than sketch-pix2ser (with KL term removed) (Chen et all P0TZ). The lightweight SketchLattice (QI ar, 202T) takes coordinates as input, making it difficult to utilize sufficient information as input. Besides, when there are noisy strokes in the sketch, inappropriate coordinates may be treated as nodes by SketchLattice and affect quality of sketch generation. Another reason is that rupted sketch instead of discarding several nodes based on probability as in (Q1 ct all 20211, which more information to be removed. Naturally, SketchLattice*. which uses the original masking method, has improved both Rec and Ret SketchHealer- 1.00 (Su et all 2020) and SketchHealer-2.. Q et arl 202Z) produce higher quality generation of corrupted sketches, reflected in the Rec metric. SketchHealer-2.0 shows an improvement on Rec compared with SketchHealer-LA which benefits from the perceptual loss to promote global semantic preservation. However, the Rel of SketchHealer-2.0 do not perform consistently between DS1 and DS2. This may be due to the fact that we reproduce the perceptual loss calculation by sketch-rnn (Ha & Eck. 2018), which is not designed for multi-category sketch generation.
On the multi-style DS2, our method still performs overall at the same level the state-
performing the lattice operation
the cor-
Qualitative Results. In Fig. E we present the sketch healing results for Pmask 10% Pmask 30% and Pmask 50%. With less loss of sketch information (Pmask 10%), the healing results for all models are relatively reasonable. However, as more strokes are covered, not all methods are effective. When Pmask 50%, some key details in the sketches are lost, e.g., the head of sheep is almost completely lost. Lmser-pix2seq successfully recreate the sheep by the limited stroke information of the head, while other methods can only produce a rough body. Observe the airplane in Fig. A When Pmast = 30% and Pmass = 50%, the shape of the fuselage in most of the generated sketches does not match the shape in the original sketch. However, the sketch recreated by Lmser-pix2seq and SketchHealer-2.0 (Qi et all 2022) maintains the structure of the fuselage
Table 1: Sketch healing retrieval performance Ret (%) 1 on DSI.
Ret
Ret
Ret
Ret @10
Ret
Ret
Model
Pmask
@10
630
@50
Pmaas
@30
@50
sketch-pix2seq
0%
31.88
48.35
58.33
30%
22.08
37.76
48.65
(Chen.etaa] 2017)
10%
28.74
45.13
55.72
50%
15.38
28.81
39.78
RPCL-piX2seq
0%
32.15
47.84
57.71
30%
17.29
29.76
40.15
Zang al. (021T
10%
26.69
41.54
52.21
50%
10.26
19.51
28.48
Sketchl.attics
0%
0.10
0.70
2.27
30%
0.07
0.42
1.57
[Q] (021
10%
0.09
0.58
2.07
50%
0.04
0.29
1.19
Sketchl.ariee
0%
0.10
0.70
-.27
30%
00.10
0.60
233
(Qr all, (222T)
10%
0.14
0.72
2.47
50%
0.06
0.41
1.54
SkeichHealer-
0%
54.29
70.42
79.22
30%
34.38
52.05
63.96
(Stletall 202()
10%
47.07
63.49
73.40
50%
21.47
38.21
51.51
Sketchflealor-200
0%
"50.83
66.50
75.31
30%
34.56
55I.76
63.20
Q 2022)
10%
43.20
59.58
69.58
50%
19.15
35.48
49.36
00%
77.20
84.82
88.71
30%
65.83
76.78
"82.59
Lmser-pix2seq
10%
73.92
82.59
87.01
50%
54.89
64.70
75.36

---

Under review as a conference paper at ICLR 2023
Table Sketch healing retrieval performance Ret (%) DS2.
Ret
Ret
Ret
Ret
Ret
Ret
Model
Pmask
@10
@30
G50
Pmnask
@10
@30
@50
sketch-pix2seq
0%
30.29
52.90
66.25
30%
17.06
36.10
52.98
(Lal. 2017))
10%
25.70
48.04
63.31
50%
10.83
25.75
41.11
RPCCI-pixxxee
0%
"34.74
56.49
7T.50
30%
12.38
28.36
45.14
Zang ct al. 20241
10%
24.77
46.22
63.83
50%
6.02
15.90
29.77
Sketchlatttice
0%
0.32
1.97
7.03
30%
0.15
1.12
4.31
@f ((21
10%
0.33
1.95
6.13
50%
0.14
1.08
3.51
SketchLattice*
0%
0.32
1.97
7.03
30%
0.33
2.07
6.58
@t (02T)
10%
0.41
2.10
6.76
50%
0.31
1.87
6.26
SketchHealer-I
0%
56.99
72.29
80.48
30%
30.06
48.84
62.54
(Su 2020)
10%
47.34
64.70
74.80
50%
16.22
33.29
48.43
SketchHccaler-.00
0%
59.98
75.90
84.14
30%
32.98
52.36
66.50
@Qi 2022)
10%
49.54
67.99
77.99
50%
18.14
35.71
51.03
0%
81.25
86.96
90.66
30%
65.94
77.65
84.22
Lmser-pix2seq
10%
78.06
83.58
88.16
50%
52.86
67.94
77.06
Table 3: Sketch healing recognition performance Rec (%) f.
Dataset
Model
Pmask
0%
10%
30%
50%
sketch-pix2seq (Chen
2012)
66.99
66.02
60.68
53.40
RPCL-pix2seq (Zang et all M121)
69.86
66.15
55.19
44.33
SketchLattice (Q1 cl all 2021)
48.88
46.57
37.87
27.91
DS1
SketchLattice* (Qi all ((2)
48.88
49.15
45.85
31.16
SketchHealer-1.0 (Su all 2020
76.76
75.05
70.47
62.86
SketchHeealer-.00 (Qi al] 2022)
77.48
75.76
71.93
64.56
Lmser-pix2seq
78.48
77.57
74.73
70.12
sketch-pix2seq (Chen all 201
88.36
87.04
82.84
75.88
RPCL-pix2seq (Zang et all 202)
90.66
87.32
76.07
63.15
SketchLattice (Q) cl all 2001
77.54
76.18
70.42
62.38
DS2
SketchLattice* (Qi all [[))
77.54
77.81
77.10
72.98
SketchHealer- et 2020
90.93
90.50
88.80
83.15
SketchHealer-2.0
(Qi et all 2/22))
92.10
91.74
90.01
84.23
Lmser-pix2seq
92.42
91.78
90.49
88.37
uniheell
10%
30%
10%
10%
30%
5005
10%
30%
50%
10%
10%
51%
10%
30%
50%
10%
30%
50%
Steah
skaidgissssa
KKCL-prcing
Seedh attice
Lasegiitio
Figure 2: The healing results with masking probabilities of 10%. 30%, and 50%.
7

---

Under review as a conference paper at ICLR 2023
better, especially the tail section. Fig. B shows the sketch healing results for different masking areas at Pmask 30%. Compared with the state-of-the-ar methods, the healing results the proposed Lmser- pisser do not differ significantly with the change of the masking location.
Full shetch
Corrupted skatch
SketchHealer 1.0
SketchFFeolar-20
Laser-pix2seg
Corrupted sketch
6
SketchHtealer 1.0
P0o
SketchHealer 2.0
Luser-pix2seg
00
Corrupted sketch
ShotchHHealees1.
ShetchHealer-220 2.0
Lmser-pix2seg
Corrupted shetch
ShetchFlealer...0 1.0
Sherchffeellar-200
Lmser-pix25eq
Figure 3: Qualitative comparisons between the proposed Lmser-pix2seq and other state of-the-art
methods. Pmosk: 30% throughout.
4.3 ABLATION STUDY
In this subsection, we examine the effectiveness of the skip connections, the number iterations T, and the image reconstruction. Our ablation experiments are conducted the larger DS1. We design five models, as shown Table of which Lmser-pix2seq (T=2) the Lmser-pix2seq model above. Table E and Table G shows the Ret and Rec results of the ablation experiments, respectively.
CNN Decoder. CAE-pix2seq with CNN decoder generates more recognizable sketches (higher Rec) compared with CE-pix2seq, This is because the CNN decoder captures the overall structure of a certain class of sketches by reconstructing the images and guides the CNN encoder to extract more accurate features (Zang all, 202I). However, the deep network makes it difficult to efficiently transfer details in sketches from the shallow encoder layers to the deep decoder layers, which results in poor Rel performance of the CAE-pix2seq.
8

---

Under review as a conference paper at ICLR 2023
Table 4: Models for ablation study,
Model
CNN encoder
CNN decoder
Skip connection
T
CE-pix2seq
Yes
No
No
CAE-pix2seq
Yes
Yes
No
Lmser-pix250 (T=1)
Yes
Yes
Yes
Lmser-pix2seq (T=2)
Yes
Yes
Yes
2
Lmser-pix2sen (T=4)
Yes
Yes
Yes
4
Table 5: Sketch healing retrieval performance Ret (%) on DS1 for ablation study.
Ret
Ret
Ret
Ret @10
Ret
Ret
Model
Pmask
@10
@30
@50
Pmask
@30
@50
0%
55.15
73.50
82.01
30%
39.55
60.90
73.57
CE-pix2seq
10%
50.13
69.85
80.07
50%
29.84
50.66
65.17
0%
48.38
64.21
73.20
30%
34.97
51.70
62.82
CAE-pix2seq
10%
43.72
60.08
69.91
50%
26.01
41.87
54.09
Lmser-pix2seq
0%
772.40
80.75
85.84
30%
38.70
70.06
77.24
(T=1)
10%
68.20
77.81
83.52
50%
47.48
60.24
68.51
Lmser-pix2seq
0%%
77.20
84.82
88.71
30%
65.83
76.78
82.39
(T=2)
10%
73.92
82.59
87.01
50%
54.89
64.70
75.36
Lmser-pix2seq
0%
77.70
84.87
88.75
30%
65.61
75.80
81.71
(T=4)
10%
74.58
82.57
86.97
50%
54.07
66.10
73.59
Table 6: Sketch healing recognition performance Rec (%) 1 on DSI for ablation study.
Dataset
Model
Pinask
0%
10%
30%
50%
CE-pix2seq
67.77
67. 67.09
64.42
60.73
CAE- CAE-piX2seq
72.38
70.66
66.60
61.60
DS1
Lmser-pix2see (T=1)
76.82
75.44
71.92
66.49
Lmser-pix2seg (T=2)
78.48
77.57
74.73
70.12
Lmser-pix2seq (T=4)
79.73
78.48
75.38
69.74
Skip Connection. The performance of Lmser-pix25st (T=1) is significantly improved over CAE- pix2seq by introducing skip connections, especially on Ret. The main reason is that while the forward connections assist the decoder in reconstructing the sketch, the feedback connections trans- mit the reconstruction information the encoder for feature map fusion, capturing the exact feature of the instance.
Number Or Iterations. The experimental results show that the performance of Lmser-pix2seg with T' = is obviously superior to that with T' 1. This indicates that one cycle does not allow the feature maps in the Lmser block to be fully fused, and the feature extracted from the sketch is not stable. When T=4, the Rec metric of Lmser Lmser-pix2seq improves overall, compared with the Lmser- pix2seq (T=2). However, the benefit of more iterations with resource consumption an issue that should be considered.
CONCLUSION
this paper, we present Lmser-pix2seq to learn stable sketch representations for sketch healing. The bi-directional skip connections in our Lmser block facilitate the fusion of the feature maps between the encoder and decoder, thus capturing the stable features of the corrupted sketch. Benefiting from the stable features, the Lmserrpix2seq global semantics. Experiments show that our approach outperforms the statc-of-the- models in the sketch healing task.
recreate sketches with high-quality and preserve their

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hervé Bourlard and Yves Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics, 59(4):291-294, 1988.
Boheng Cao, Shikui Tu, and Lei Xu. Flexible-clmser: Regularized feedback connections for biomedical image segmentation. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 829-835. IEEE, 2021.
Yajing Chen, Shikui Tu, Yuqi Yi. and Lei Xu. Sketch-pix2seqq model to generate sketches of 2017.
multiple categories. arXiv preprint
Ayan Das, Yongxin Yang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song. Cloud2eurve: Generation and vectorization of parametric sketches. In Proceedings of IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7088-7097, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint Stt::20000.1129,, 2020.
songwei Ge, Vedanuj Goswami, Larry Zitnick, and Devi Parikh. Creative sketch generation. In International Conference Learning Representations, 2021.
Yuze Guo, Wenjing Huang, Yajing Chen, and Shikui Tu. Regularize network skip connections by gating mechanisms for electron microscopy image segmentation. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 868-873. IEEE, 2019.
David Ha and Douglas Eck. A neural representation sketch drawings. In International Conference Learning Representations, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Huimin Huang, Lanfen Lin, Ruofeng Tong, Hongjie Hu, Qiaowei Zhang, Yutaro Iwamoto, Xianhua Han, Yen-Wei Chen, and Jian Wu. Unet 3+: full-scale connected unet for medical image segmentation. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1055-1059. IEEE, 2020a.
Wenjing Huang. Shikui Tu, and Lei Xu. Deep based Imser and strengths of two built-in dualities. Neural Processing Letters, pp. 1-17, 2020b.
2020,
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations, 2013.
Peiying Li, Shikui Tu, and Lei Xu. Gan flexible Imser for super-resolution. In Proceedings of the 27th ACM International Conference Multimedia, pp. 756-764, 2019.
Hangyu Lin, Yanwei Fu, Xiangyang Xue, and Yu-Gang Jiang. Sketch-bert: Learning sketch bidi-
rectional encoder representation from transformers by self-supervised learning of sketch gestalt. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp-
7768, 2020.
10

---

Under review as a conference paper at ICLR 2023
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural net- works for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pp. 565-571. IEEE, 2016.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Yonggang Qi, Guoyao Su, Pinaki Nath Chowdhury, Mingkang Li, and Yi-Zhe Song. Sketchlattice: Latticed representation for sketch manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 953-961, 2021.
Yonggang Qi, Guoyao Su, Qiang Wang. Jie Yang, Kaiyue Pang, and Yi-Zhe Song. Generative sketch healing. International Journal of Computer Vision, pp. 1-16, 2022.
Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, and Moacir Ponti. Sketchformer: Transformer-based representation for sketched structure. In Proceedings of the IEEE/CVE con- ference on computer vision and pattern recognition, pp. 14153-14162, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation, In International Conference on Medical image computing and computer- assisted intervention, pp.
Springer, 2015.
Jifei Song. Kaiyue Pang, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales. Learning to sketch with shorteut cycle consistency. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 801-810, 2018.
Guoyao Su. Yonggang Qi, Kaiyue Pang, Jie Yang. and Yi Zhe Song. Sketchhealer graph-to- sequence network for recreating partial human sketches. In British Machine Vision Conference, 2020.
Tiange Xiang, Chaoyi Zhang, Dongnan Liu, Yang Song, Heng Huang, and Weidong Cai. Bio-net: learning recurrent bi-directional connections for -ecoddr-decoder architecture. In International conference on medical image computing and computer-assiste intervention, pp. 74-84. Springer, 2020.
Lei Xu. Least mse reconstruction by self-organization. i. multi-layer neural-nets & ii. further the- oretical and experimental studies on one-layer nets. In /Proceedings) 1991 IEEE International Joint Conference on Neural Networks, pp. 2362-2367. IEEE, 1991.
Lei Xu. Least mean square error reconstruction principle for self-organizing neural-nets. Neural Networks, 6(5):627-648, 1993.
Lei Xu. An overview and perspectives bidirectional intelligence: Lmser duality, double ia har- mony.and causal computation. IEEE/CAA Journal of Automatica Sinica, v.6(04):432,, 2019.
Qian Yu, Yongxin Yang. Yizhe Song, Tao Xiang, and Timothy M Hospedales. Sketch-a-net that beats humans. British Machine Vision Conference (BMVC), 2015.
Sicong Zang, Shikui Tu, and Lei Xu. Controllable based sketch synthesis from self- organized latent space. Neural Networks, 137:138-150, 2021.
Tao Zhou, Chen Fang, Zhaowen Wang, Jimei Yang, Byungmoon Kim, Zhili Chen, Jonathan Brandt, and Demetri Terzopoulos. Learning to doodle with deep networks and demonstrated strokes. In British Machine Vision Conference, volume 1, pp. 4, 2018a.
Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: A nested u-net architecture for medical image segmentation. In Deep learning in medical image analysis and multimodal learning for clinical decision support, pp. 3-11. Springer, 2018b
11

---

Under review as a conference paper at ICLR 2023
A THE DIFFERENCES BETWEEN SKETCH-PIX2SEQ RPCL-PIX2SEQ AND LMSER-PIX2SEQ
sketch-pix2seg RPCI--pix2sq and Lmser-pix2seq differ in the problems they address and in the design of the network architecture. The sketch-pix2seq is concerned with category-leve genera- tion, which leads to a high Rec score. The main function of RPCL-pix2seq is to generate sketches according given categories instance-level, with the strength of high Ret score. For model design, sketch-pix2seq uses a CNN encoder to extract features and RNN decoder output recreated sketches. RPCL-pix2seq in- troduces a CNN decoder on top of sketch-pix2seq and assumes the latent space follows a Gaussian mixture model (GMM). The CNN decoder in RPCI-pix2seq plays role of regularization and helps the formation of feature concepts. The network architecture of RPCL- RPCL-pix2seq is similar to the CAE- pix2seq model in section 4.3. Lmser-pix2seq constructs bi-directional skip connections between the CNN encoder and CNN decoder. This allows both the CNN encoder and the CNN decoder to be involved feature extraction and image reconstruction. Loops obtained based on bi-directional skip connections are also discussed. Experimental results show that two and more cycles are effective in capturing sketch features.
styles. Lmser-pix2seq focuses capturing features at the sketch
B SKETCH HEALING RESULTS FOR IRREGULAR HOLE MASKING
GT
Mask
Input
Results
df
m
uun
ruu
m
Q
80
pO
9.0
100
3X4
Omog
Figure 4: Sketch healing results for irregular hole masking. The The left three columns in the figure are ground truth sketches, masks and masked sketches. The right six columns in the figure are the sketches recreated by Lmser-pix2seq.
C RETRIEVAL METRIC
Fig. E shows two retrieval methods, and the metric used in this paper correspond to (a) in the figure.
Compared with the retrieval metric in SketchHealer-11 1.0 etll 2020, Ret is more challenging. For each query, Ret allows only one correct result, while hundreds of answers exist for the other method.
12

---

Under review as a conference paper at ICLR 2023
Latent Space
RNN Decoder
Eneoder
Cornupted Sketch
Rustence
Correspanding
Full Sketch &
Recreated
Sketch
(a)
Latent Space
Encoder
Same Tabel Gallery Full Sketch
Query Corrupted
Sketch X
(b)
Figure 5: Two types of retrieval metrics. The retrieval is performed in the latent space. Blue circles represent queries and red circles represent samples with correct retrieval results. (a) The retrieval method from RPCL-pix2seq (Zang et all 2021), e. Rel, an instance-level approach. (b) The retrieval method from SketchHeeler-1100 all 2020) is category-level approach.
D LMSER-PIX2SEQ ONLY TRAINED WITH CNNS
When Lmser-pix2seg only trained with CNNs, sketch healing then turned into image inpainting. Table @ and Table N report the relevant metrics. Fig. @ shows the results of some sketch reconstruc- tions. Although these images maintain the overall structure of the sketch well, some of the results are blurry and incomplete. This is why an RNN decoder is needed to generate the sequences.
Table 7: Sketch image inpainting recognition performance Rec (%) f.
Dataset
Model
Pmask
0%%
10%
30%
50%
DS1
CNNs
89.28
87.19
80.56
71.07
DS2
CNNS
96.10
95.60
94.33
92.12
13

---

Under review as a conference paper at ICLR 2023
Table Sketch image inpainting retrieval performance Ret (%) f.
Ret
Ret
Ret
Ret
Ret
Ret
Model
Pmosk
@ @10
@30
550
Pmask
@10
@30
@50
0%
98.81
99.38
99.55%
63.45
73.78
79.76
CNNs DSI
10%
87.33
92.00
94.17
50%
44.30
56.37
64.65
0%
97.73
98.52
98.925 08.92%
35.77
67.81
75.00
CNNs DS2
10%
82.74
88.46
91.22
50%
35.48
48.65
58.20
Pmack 0.0
Pmaak 0.1
Pmask 0.3
x
3
Pmuas 0.5
$
Figure 6: When only CNNs is trained, some results of CNN decoder output.
14


Reviews:
[{"review_id": "tskvBul9_gX", "reviewer": "Reviewer_rRmX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is an attempt to solve the problem of ‘sketch healing’; trying to recover the original sketch from the corrupted one. The authors propose Lmser-pix2seq, which consist of Lmser block (an encoder-decoder for healing rasterized sketch) and an LSTM decoder (to convert from latent embedding from lmser to full sequence). The evaluation is done on 1) if the recover sketch can be classified as the same class as the complete sketch and 2) if the embedding of the recovered sketch can be used to retrieve the full sketch. The results are better than previous works on both metric, however only slightly in the recognition test.", "summary_of_the_review": "While I like the cleverness of the approach, I’m a bit concern about the actual problem and its usefulness. Sure sketch healing is great for recover the missing details of the sketch, but in the current setting this is only applying to this specific corruption method. I think it would be great to have another section about generalization for example how this can be applied to actual sketch corruption various source, or how much will this help if we use it as data augmentation method for sketch related task. R_rec sort of try to cover this by showing how it can help with the sketch classification task, but the proposed method only gain small amount of improvement. So as it is currently I’m a bit hesitant to recommend acceptance. \n\nOthers:\nAlso, all these discussion about sketch corrupting, sketch healing, etc. seems like it would be a natural extension to apply diffusion model to it. ", "strength_and_weaknesses": "Strength\n - The self supervised manner of training is pretty clever. \n - The learning of y from the mini task of healing rasterized sketch is also interesting\n - The output is a sequence S of sketch, so should be more useful than rasterized output\n - The qualitative result comparison looks great and shows the strength of the proposed method against others.\n\nWeakness\n - The 2 quantitative metrics used aren’t really satisfying. The R_ret comes from feeding the rasterized version of the generated sketch and the full sketch into the proposed encoder, which was trained for this method to match the two. It doesn’t seem fair to other methods when comparing them. The R_rec is less problematic but the improvement of the proposed method over other methods is very small. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "nHaooR2qvz", "reviewer": "Reviewer_po7R", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new solution for the task of Sketch healing, where corrupted or unfinished sketches are reconstructed without the corruption. This task is commonly accomplished through representation learning. The manuscript describes a method focused on ‘stable representations’, where complete and corrupted sketches should have similar representations; this is a concept not dissimilar to the principles of representation learning in self-supervision models. To achieve such stability the model is composed of a CNN-based autoencoder with least mean square error reconstruction (Lmser) blocks that reconstruction raster sketches plus an RNN-based decoder that reconstructs vector sketches. The manuscript also has empirical evidence supporting this new model as the State-of-the-Art on sketch healing.\n", "summary_of_the_review": "The paper applies the existing Lmser AEs method to the task of learning stable representations for sketches, with the final goal of performing sketch healing. The authors combine the Lmser AE with the usual recurrent decoder for vector sketch generation. While there are flaws with one of the baseline comparisons and the ablations could be more thorough at figuring out why this model works, the performance gains are significant and I believe this model would be a good addition to the sketch community’ literature. ", "strength_and_weaknesses": "The strength is in the core contribution of the paper: the choice of using the Lmser blocks is well motivated by the need for stable representations for complete and corrupted sketches; the idea is new to the sketch literature; the empirical results show the effectiveness of using this new representation combined with the traditional recurrent decoder.\n\nFor weaknesses, I’ve found the comparison agains the SketchLattice baseline to be flawed. The input requirements for that model should have been followed to allow for fair comparison. Furthermore, other competing models such as SketchHealer-2.0 have presented results aligned with the original SketchLattice papers in their comparisons. I am also curious as to how well the model would work when only trained with the CNNs, without the vector decoder. I understand the RNN decoder to be essential for the output to not be blurry, but is it also essential for obtaining a stable representation?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ltr7CzgEN2", "reviewer": "Reviewer_oNd8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the sketch healing problem. In other words, it addresses the problem of removing corruptions from sketches. In this work, the authors have proposed the least mean square error reconstruction (LMSER) block for the same, which essentially falls within the encoder-decoder paradigm. The model takes a corrupted sketch as input, then the LMSER encoder computes the embeddings of structural patterns of the input, while the decoder reconstructs the original sketch from the latent embeddings. The decoder is constructed by a recurrent neural network which recreates the sketches from the features captured by the LMSER block.\n", "summary_of_the_review": "The paper combines two existing models for sketch healing task, whose novelty is incremental.", "strength_and_weaknesses": "**Strengths**\n(1) The proposed model has reported state-of-the-art performance on existing datasets for the sketch healing tasks.\n\n**Weaknesses**\n(1) The proposed model combines LMSER and Pix2seq models for the sketch healing task, which is incremental. Furthermore, the difference from sketch-pix2seq and RPCL-pix2seq is not clear. I understand that the latter models have been proposed for sketch generation, but joining them with a sketch encoder is not really difficult.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "csa1S5tqsS1", "reviewer": "Reviewer_hWp5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes LMSER-PIX2SEQ for sketch healing. The method is an encoder-decoder similar to VAE and performs very good with respect to the other baselines.   ", "summary_of_the_review": "I think this study is a much better fit for a computer vision conference. \n", "strength_and_weaknesses": "**Strength**\n\nThe paper is well-written and the method shows great performance in the experiments.\\\n\n\n**Weaknesses**\n- The method's novelty is limited.\n- I'm not familiar with the literature but I think the paper might be a better fit for a computer vision conference (like the other baselines).\n- In section 4.1 there is a discussion about the information leakage in the dataset preparation in sketchhealer.1 and also changing the Ret evaluation from what the other baselines have previously reported on. How much of the performance gap shown in tables 1 and 2 is coming from these factors, especially the updated Ret metric?\n- typo: page 2, the second paragraph above the Related Works, \"oin\" should be \"in\"\n\n- Question: Why do the authors think that their method is beating the other baselines with a big margin in Ret ( table 2) but getting closer numbers in Rec (table 3) ?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject