Title: Prune and Repaint: Content-Aware Image Retargeting for any Ratio
Venue: NeurIPS 2024
Accepted: yes
Keywords: Image Retargeting

content-aware seam-carving

adaptive repainting
Abstract: Image retargeting is the task of adjusting the aspect ratio of images to suit different display devices or presentation environments. However, existing retargeting methods often struggle to balance the preservation of key semantics and image quality, resulting in either deformation or loss of important objects, or the introduction of local artifacts such as discontinuous pixels and inconsistent regenerated content. To address these issues, we propose a content-aware retargeting method called PruneRepaint. It incorporates semantic importance for each pixel to guide the identification of regions that need to be pruned or preserved in order to maintain key semantics. Additionally, we introduce an adaptive repainting module that selects image regions for repainting based on the distribution of pruned pixels and the proportion between foreground size and target aspect ratio, thus achieving local smoothness after pruning. By focusing on the content and structure of the foreground, our PruneRepaint approach adaptively avoids key content loss and deformation, while effectively mitigating artifacts with local repainting. We conduct experiments on the public RetargetMe benchmark and demonstrate through objective experimental results and subjective user studies that our method outperforms previous approaches in terms of preserving semantics and aesthetics, as well as better generalization across diverse aspect ratios. Codes will be available at
 https://github.com/fhshen2022/PruneRepaint.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an innovative method for image retargeting. It addresses two core challenges in image retargeting (preserving the main information and avoiding artifacts on key objects) simultaneously by carefully devising a content-aware seam-carving method and an adaptive repainting method respectively. Both quantitative and qualitative results demonstrate the advantages of the method over others.\n\nSTRENGTHS:\n1. Significant improvement over prior works on both preserving semantic completeness and maintaining the harmony of the retargeted contents. \n2. The paper is well-motivated and the method presented is well-structured. \n3. The proposed idea is exciting. The semantic-guided local repainting solution is very reasonable and inspirable for the following researchers on this topic. The two key designs are effective.  \n4. The paper is well organized and written, with clear explanations of the method and effective visual representations.\n\nWEAKNESSES:\n1. Detailed explanations for some key concepts are missing and adding them will further enhance the clarity of this paper, e.g., in Line 142: the process for computing x_0 should be presented. Is it the saliency center of the entire image or the saliency center of a row or column?\n2. It will be better for this paper to demonstrate its advantages if the authors include recent deep learning works related to image editing/generation for comparison, especially those global generation methods. As stated, the proposed method selectively regenerates the abrupt pixels and preserves the foreground consistency and local smoothness over previous global generation methods without maintaining foreground consistency with the original image. Although Figure 1 shows an example, more comparisons will be highly beneficial to verify such advantages.\n3. The authors perform subjective evaluation by designing a User Study Metric. However, it lacks explanations with corresponding visual samples to show the complementarity among the four different evaluation metrics in Table 2. For example, the authors should include some samples with different degrees of deformation or distortion for comparison and explanation. \n4. The ARRD module in Figure 2 is not very clear to show the determination process for inpainting or outpainting. Expanding the mask map seems an optional choice and only works for some special cases when the foreground regions are large and direct cropping will lose too much object contents. However, in Figure 2, it appears always required for any input and ratio.\n\nQUESTIONS:\n1.\tFrom Figure 6, in addition to addressing the drawback of BR, which is unable to handle discontinuities in foreground pixels, I also notice that the inclusion of AR can more faithfully restore the depth and proportion of the original image (such as the size relationship between the cylinder and the background building in architectural images). Can the authors briefly discuss the reasons behind this advantage?\n2.\tI am very interested in the performance on videos. Intuitively, the proposed method will work well on each frame. However, inter-frame smoothness will be a new problem for videos. Without explicit inter-frame consistency regularization, will the proposed model result in cross-frame mutation and jitter?  \n3.\tIn line 266, what is the detailed difference between BR and AR. Both of them regenerate background regions, but what are the reasons that make BR unable to address discontinuities in foreground pixels?\n4.\tFrom Table 1, I notice that the performance for different target ratios varies a lot. Please explain this difference. \n5.\tIn line 178, how to decide the threshold η?\n\nLIMITATIONS:\nLimitations and analyses are included by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new image retargeting framework that prunes background and repaints local connections. It improves the traditional seam-carving with semantic guidance to make the pruning content-aware, avoiding deformation and loss of important objects. Meanwhile, the authors introduce an adaptive repainting module using an image-conditioned diffusion model to selectively inpaint or outpaint local regions to achieve local smoothness. The authors also design two evaluation metrics, including the Saliency Discard Ratio (SDR) and a user study metric for evaluation. The proposed method shows superior performance over others and generalization to varying target ratios.\n\nSTRENGTHS:\nGood motivation and key problems are clearly summarized.\n\nThe method is technically sound and novel. I am very interested in this task and like the proposed idea. It appears to be a pioneering work in the image retargeting community using diffusion models. \n\nImpressive results and a large improvement over previous works in preserving object completeness, and coherence, and it holds better generalization.\n\nWEAKNESSES:\nAs pointed out by the authors, the inference speed has a large room to be improved. I suggest the authors adopt some accelerated diffusion models to improve it.\n\nThe authors argue that their method can work well on any ratio, such as 1:1 and 4:3, yet these visual results are not illustrated.\n\nMinor writing or grammar mistakes, such as missing the point in line 70, and missing “the” in the caption of Figure 3.\n\nHow does the text prompt work? I am interested in its contributions to the retargeting results.\n\nQUESTIONS:\nIn equation (5), what does W_s denote? Is it the same as W in equation (2), meaning the width of the input image?\n\nFor the proposed metric SDR, the authors use the width change of the saliency map to measure semantic completeness. How to implement this measurement? If my understanding is correct, is it to sum the width of all lines in the saliency map and output the maximum one as the width of the saliency map? Why do not use the metrics in salient object detection such as F-measure or use IOU to evaluate?\n\nIn Table 2, I think content completeness and aesthetic scores are more important than the other two scores as they measure the two key views for image targeting. So, I wonder whether there is a better strategy to obtain an overall score rather than just averaging them.\n\nLIMITATIONS:\nThe authors explicitly discussed the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work contributes to a new image retargeting model named PrueRepaint, which is adaptive to work with any target ratio. \n\nThe authors first improve the traditional seam-carving method with saliency priors to achieve content-aware pruning and protect important semantic regions. \n\nAfter that, they introduce an adaptive repainting module using the diffusion model to maintain local smoothness after pruning. \n\nThe method is unique compared to the previous and also effective. The newly proposed metrics are reasonable. Experiments have demonstrated the effectiveness of the key designs and the large advantages over other methods.\n\nSTRENGTHS:\n- I believe the view that the authors address the image retargeting task is novel and very important. This work takes a step towards a spatial-variable diffusion model. In contrast, most diffusion models focus on the spatial-fixed community. \n\n- The method is well-designed and reasonable. The proposed idea differs a lot from previous cropping-based or global generation methods. The authors provide a more reasonable and effective solution path.\n\n- The authors also introduce two reasonable evaluation metrics.\n\n- Ablation studies verify the effectiveness of each design of the proposed method. \n\n- The proposed method achieves fairly superior results compared to previous models.\n\nWEAKNESSES:\n- It will be more convincing to verify the superiority of the proposed method if more methods such as the ‘InGAN’ model in Figure 1 can be involved for comparison. The previous methods address this task from varying views, such as using cropping, scaling, seam-carving, and a generative model. This work seems to provide a new idea. So I expect a more comprehensive comparison to other solutions to show its advantages.  \n\n- Although the proposed method has achieved large improvement over previous works on avoiding artifacts, the results still hold some distorted regions. In my view, strengthing local correlations will be helpful to solve this problem. \n\n- The dataset contains limited samples. Although the authors try to output different ratios and the results show the advantages and better generalization over previous works, it will be much better to enhance the results and comparisons with more samples. So, I suggest the authors collect a new high-quality dataset with diverse scenarios for this task and test the model with a larger dataset for evaluation in the future.\n\nQUESTIONS:\n- The authors point out that ‘the repainting region generated by ARRD is not complete enough and contains certain distorted regions’ in the limitation section. Please give more explanations for this limitation.\n\n- The result of ‘+CSC’ in Table 3 and ‘background repainting’ in Table 4 hold the same score. This should not be a coincidence, please explain the reason behind it.\n\n- For the IP-adapter, it seems that the proposed method does not rely on text prompts to fulfill or improve the retargeting result. However, I am still interested in what text prompt is used in your design. Have you tried different text prompts such as “keep key semantics” and how about the results?  \n\n- The output ratio is set to 16:9, 1:1, 4:3, and 9:16. What are the reasons for selecting these ratios as the target output? In previous image retargeting works, I also notice some other ratios such as 2:3 and other extreme ratios. \n\n-The repainting choice that inpainting or outpainting is based on setting a hyperparameter and decided by comparing the target ratio and the foreground size. I expect an adaptive strategy for this determination. Can you provide some potential solutions for this problem?\n\nLIMITATIONS:\nThe authors discussed the limitations adequately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The work presents an addon using Diffusion models to Seam Carving to perform Content Aware resizing of images.

STRENGTHS:
+ None

WEAKNESSES:
- The work seems to be a rehash of Seam Carving, and some diffusion models were added to perform retargeting. There is no novelty in the method.

- One page of the paper is direct equations from Seam Carving. How can it be a contribution?

- The application is quite dated and has not made any significant through the methods proposed, which are logical but lack impact.

- The writing is hazy, and there is barely any motivation about the approach and the application.

QUESTIONS:
Please see above.

LIMITATIONS:
Limitations not well discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2024-07-18