Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper addresses the sketch healing problem. In other words, it addresses the problem of removing corruptions from sketches. In this work, the authors have proposed the least mean square error reconstruction (LMSER) block for the same, which essentially falls within the encoder-decoder paradigm. The model takes a corrupted sketch as input, then the LMSER encoder computes the embeddings of structural patterns of the input, while the decoder reconstructs the original sketch from the latent embeddings. The decoder is constructed by a recurrent neural network which recreates the sketches from the features captured by the LMSER block.

strength_and_weaknesses: **Strengths**
(1) The proposed model has reported state-of-the-art performance on existing datasets for the sketch healing tasks.

**Weaknesses**
(1) The proposed model combines LMSER and Pix2seq models for the sketch healing task, which is incremental. Furthermore, the difference from sketch-pix2seq and RPCL-pix2seq is not clear. I understand that the latter models have been proposed for sketch generation, but joining them with a sketch encoder is not really difficult.
clarity,_quality,_novelty_and_reproducibility: This paper is well-written. It's clear and easy to follow, but the novelty may be limited.
summary_of_the_review: The paper combines two existing models for sketch healing task, whose novelty is incremental.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper is an attempt to solve the problem of ‘sketch healing’; trying to recover the original sketch from the corrupted one. The authors propose Lmser-pix2seq, which consist of Lmser block (an encoder-decoder for healing rasterized sketch) and an LSTM decoder (to convert from latent embedding from lmser to full sequence). The evaluation is done on 1) if the recover sketch can be classified as the same class as the complete sketch and 2) if the embedding of the recovered sketch can be used to retrieve the full sketch. The results are better than previous works on both metric, however only slightly in the recognition test.
strength_and_weaknesses: Strength
 - The self supervised manner of training is pretty clever. 
 - The learning of y from the mini task of healing rasterized sketch is also interesting
 - The output is a sequence S of sketch, so should be more useful than rasterized output
 - The qualitative result comparison looks great and shows the strength of the proposed method against others.

Weakness
 - The 2 quantitative metrics used aren’t really satisfying. The R_ret comes from feeding the rasterized version of the generated sketch and the full sketch into the proposed encoder, which was trained for this method to match the two. It doesn’t seem fair to other methods when comparing them. The R_rec is less problematic but the improvement of the proposed method over other methods is very small. 
clarity,_quality,_novelty_and_reproducibility: The paper is generally well written.
Would probably need the full code release from the authors to ensure reproducibility.
summary_of_the_review: While I like the cleverness of the approach, I’m a bit concern about the actual problem and its usefulness. Sure sketch healing is great for recover the missing details of the sketch, but in the current setting this is only applying to this specific corruption method. I think it would be great to have another section about generalization for example how this can be applied to actual sketch corruption various source, or how much will this help if we use it as data augmentation method for sketch related task. R_rec sort of try to cover this by showing how it can help with the sketch classification task, but the proposed method only gain small amount of improvement. So as it is currently I’m a bit hesitant to recommend acceptance. 

Others:
Also, all these discussion about sketch corrupting, sketch healing, etc. seems like it would be a natural extension to apply diffusion model to it.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes LMSER-PIX2SEQ for sketch healing. The method is an encoder-decoder similar to VAE and performs very good with respect to the other baselines.   
strength_and_weaknesses: **Strength**

The paper is well-written and the method shows great performance in the experiments.\


**Weaknesses**
- The method's novelty is limited.
- I'm not familiar with the literature but I think the paper might be a better fit for a computer vision conference (like the other baselines).
- In section 4.1 there is a discussion about the information leakage in the dataset preparation in sketchhealer.1 and also changing the Ret evaluation from what the other baselines have previously reported on. How much of the performance gap shown in tables 1 and 2 is coming from these factors, especially the updated Ret metric?
- typo: page 2, the second paragraph above the Related Works, "oin" should be "in"

- Question: Why do the authors think that their method is beating the other baselines with a big margin in Ret ( table 2) but getting closer numbers in Rec (table 3) ?
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and the method shows very good performance. \
The novelty of the method is a bit limited. Regarding reproducibility, I think that the draft is clear enough for reproducing the method but I encourage the authors to release the code, the new Ret metric, and the revised data feeding mentioned in section 4.1.  
summary_of_the_review: I think this study is a much better fit for a computer vision conference.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a new solution for the task of Sketch healing, where corrupted or unfinished sketches are reconstructed without the corruption. This task is commonly accomplished through representation learning. The manuscript describes a method focused on ‘stable representations’, where complete and corrupted sketches should have similar representations; this is a concept not dissimilar to the principles of representation learning in self-supervision models. To achieve such stability the model is composed of a CNN-based autoencoder with least mean square error reconstruction (Lmser) blocks that reconstruction raster sketches plus an RNN-based decoder that reconstructs vector sketches. The manuscript also has empirical evidence supporting this new model as the State-of-the-Art on sketch healing.

strength_and_weaknesses: The strength is in the core contribution of the paper: the choice of using the Lmser blocks is well motivated by the need for stable representations for complete and corrupted sketches; the idea is new to the sketch literature; the empirical results show the effectiveness of using this new representation combined with the traditional recurrent decoder.

For weaknesses, I’ve found the comparison agains the SketchLattice baseline to be flawed. The input requirements for that model should have been followed to allow for fair comparison. Furthermore, other competing models such as SketchHealer-2.0 have presented results aligned with the original SketchLattice papers in their comparisons. I am also curious as to how well the model would work when only trained with the CNNs, without the vector decoder. I understand the RNN decoder to be essential for the output to not be blurry, but is it also essential for obtaining a stable representation?

clarity,_quality,_novelty_and_reproducibility: Writing can be improved specifically on the Introduction. More notes will follow at the end of this comment. 

I believe this is a paper with a single contribution, which is the application of Lmser to the task of Sketch Healing. It is focused and delivers on the experimental results; that said I find it is not necessary to break this single contribution into three which are each derived from the previous one.

In the related work section, I find it odd that the authors define reconstruction as “conditional generation” and class-conditional generation as “unconditional generation”. The task of sketch healing is similar to the one of reconstructing images from noisy inputs and unless there is a pattern in the sketch literature for this naming convention it would be better to follow the general computer vision conventions.

The authors mention the LSTM model is referenced from Sketch-RNN, does that include the GMM model at the output?

In section 3.1, the manuscript mentions that corrupted sketches are obtained by “randomly rounding off a certain percentage of strokes”, which would make corrupt sketches always be the same as unfinished sketches. It is not clear however when this corruption technique is used since in section 4.1 another common method is described to be applied for the experiments. Was the first method used for training the model and the second used for testing?

The naming and explanations of the metrics Rec and Ret could be amended to better explain what are the actual metrics used. Is Rec simple top-1 accuracy? Is Ret precision@K? The short names can be used on the tables but please make meaning explicit within the text.

Extra notes:
- Sec 1. “due to the difficulty in learning” can be removed from the abstract and introduction
- Sec 1. “it is not always available” needs a better connective “but this information” would work
- Sec 1. “SketchLattice is a lightweight model which limits its performance” is not always a true statement for lightweight models. 
- Sec 1. “Different to […] we expect the network to learn stable sketch representations”. This motivator does not explain why a cnn would be better than a gnn at learning stable representations
- Sec 1. “and even more” should be “and worse” or similar
- Sec 1. “oin” should be “in”
- Sec 3.3. “raseter”

summary_of_the_review: The paper applies the existing Lmser AEs method to the task of learning stable representations for sketches, with the final goal of performing sketch healing. The authors combine the Lmser AE with the usual recurrent decoder for vector sketch generation. While there are flaws with one of the baseline comparisons and the ablations could be more thorough at figuring out why this model works, the performance gains are significant and I believe this model would be a good addition to the sketch community’ literature.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary**: The paper is an attempt to solve the problem of ‘sketch healing’; trying to recover the original sketch from the corrupted one. The authors propose Lmser-pix2seq, which consist of Lmser block (an encoder-decoder for healing rasterized sketch) and an LSTM decoder (to convert from latent embedding from lmser to full sequence). The evaluation is done on 1) if the recover sketch can be classified as the same class as the complete sketch and 2) if the embedding of the recovered sketch can be used to retrieve the full sketch.

**Strengths**: The paper is well-written and the method shows great performance in the experiments.

**Weaknesses**: The following weaknesses were identified by the reviewers:

1. incremental novelty - combining LMSER and Pix2seq. [oNd8, hWp5]
2. the difference between sketch-pix2seq and RPCL-pix2seq is unclear. [oNd8]
3. The R_ret metric is not fair -- the generated sketch and full sketch are compared using the model encoder. [rRmX]
4. The results using R_rec metric show only small improvements. [rRmX]
5. Only applied to a specific corruption method. How does the method generalize to actual sketch corruption from various sources? [rRmX]
6. how much of the performance improvement is related to the change in the Ret metric and the information leakage? [hWp5]
7. Why are the improvements much better for Ret metric (table 2) than the Rec metric (table 3)? [hWp5]
8. comparison with SketchLattice is flawed (input requirement is not the same). [po7R]
9. Other models presented results aligned with the original SketchLattice paper. [po7R]
10. How well does it work when only trained with CNNs (without vector decoder)? [po7R]
11. various improvements in presentations [po7R]
12. unclear if the "unfinished sketch" corruption is actually used. [po7R]

**Discussion:** During the discussion, one reviewer remained positive about the paper, while others did not change their ratings. The major issues were 1) the incremental novelty, and 2) the suitability of the proposed retrieval metric. Regarding novelty, the paper mainly combines LMSER and pix2seq. Regarding the retrieval metric, the AC pointed out that the Ret metric uses each model's own encoder to do the evaluation, which means that each compared method is using a different retrieval model. This doesn't seem fair because some models may have better encoders (more complex architectures), or be specifically trained to have a retrievable internal representation. To be fair, the same retrieval model (distinct from all the sketch reconstruction methods) should be used for all methods to evaluate Ret. It was also pointed out that using a CNN (as asked by Reviewer po7R) actually outperformed the proposed method for small mask percentages on retrieval, and outperforms all methods at Rec. Reviewer po7R pointed out that using CNN might not be suitable for sketch healing, since the results will be blurry. Probably more suitable evaluation metrics are needed. 

justification_for_why_not_higher_score: The novelty is incremental, and evaluation metrics need to be updated to be more fair. 
justification_for_why_not_lower_score: n/a
Decision: Reject