Title: Synthesize Privacy-Preserving High-Resolution Images via Private Textual Intermediaries
Venue: NeurIPS 2025
Accepted: yes
Keywords: differential privacy

private evolution

private text bridges
Abstract: Generating high-fidelity, differentially private (DP) synthetic images offers a promising route to share and analyze sensitive visual data without compromising individual privacy. However, existing DP image synthesis methods struggle to produce high-resolution outputs that faithfully capture the structure of the original data. In this paper, we introduce a novel method, referred to as Synthesis via Private Textual Intermediaries (SPTI), that can generate high-resolution DP images with easy adoptions. The key idea is to shift the challenge of DP image synthesis from the image domain to the text domain by leveraging state-of-the-art DP text generation methods. SPTI first summarizes each private image into a concise textual description using image-to-text models, then applies a modified Private Evolution algorithm to generate DP text, and finally reconstructs images using text-to-image models. Notably, SPTI requires no model training, only inferences with off-the-shelf models. Given a private dataset, SPTI produces synthetic images of substantially higher quality than prior DP approaches. On the LSUN Bedroom dataset, SPTI attains an FID $=$ 26.71 under $\epsilon=1.0$, improving over Private Evolution’s FID of 40.36. Similarly, on MM-CelebA-HQ, SPTI achieves an FID $=$ 33.27 at $\epsilon=1.0$, compared to 57.01 from DP fine-tuning baselines. Overall, our results demonstrate that Synthesis via Private Textual Intermediaries provides a resource-efficient and proprietary-model-compatible framework for generating high-resolution DP synthetic images, greatly expanding access to private visual datasets. Our code release: https://github.com/MarkGodrick/SPTI


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SPTI (Synthesis via Private Textual Intermediaries), a novel framework for privacy-preserving differentially private (DP) image synthesis. The method first captions private images using a vision-language model (VLM), then applies an Augmented Private Evolution algorithm to generate DP captions, and finally generates synthetic images from the DP text via a diffusion model. Notably, the entire pipeline is training-free and capable of producing high-resolution outputs. Experiments show favorable performance based on FID scores. The authors also contribute a new dataset, Sprite Fright, for benchmarking.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea of using text as an intermediary for privacy-preserving image synthesis is both novel and practical, particularly because it avoids the need for model training and leverages powerful pretrained models.\n\n2. The proposed Aug-PE algorithm for DP text generation is a meaningful technical contribution.\n\n3. The authors contribute a new dataset, Sprite Fright, which could benefit future work in DP image synthesis.\n\nWeaknesses:\nMajors:\n1. Information loss via captioning is underexplored.\nUsing VLMs to extract captions from images can introduce significant information loss, especially in specialized domains or when fine-grained details are involved. While this is briefly acknowledged in the limitations section, it deserves deeper empirical investigation. The current evaluation relies solely on the FID metric, which does not capture semantic fidelity. Including downstream tasks (e.g., classification) would better evaluate the utility of the generated images.\n\n2. Missing results for DP-finetuning.\nThe experiments section mentions a comparison against DP-finetune, but Table 1 does not report these results. This inconsistency should be addressed.\n\n3. Choice of similarity metric.\nSection 3.2 states that Euclidean distance is used for nearest-neighbor search based on embeddings. However, cosine similarity is more commonly used in text/image embedding spaces. Justifying this choice—or comparing both—would strengthen the methodology.\n\nMinors:\n1. The last paragraph of the Introduction lacks a summary of Section 5. Adding this would improve flow and completeness.\n2. Inconsistent naming of prior work—please standardize references to \"PE\" or \"PE image\" throughout the paper.\n\nQUESTIONS:\n1. Why are no downstream tasks included to assess the semantic utility of the synthesized images? (Related to Major Weakness 1)\n\n2. Given the known generalization limitations of pretrained models, can you provide examples or analysis of failure cases—i.e., images whose captions omit critical visual information? (Also related to Major Weakness 1)\n\n3. Why are the DP-finetuning results mentioned in the text but not included in Table 1? (See Major Weakness 2)\n\n4. Why was Euclidean distance selected for similarity search in embedding space, instead of cosine similarity? (See Major Weakness 3)\n\n5. The FID metric has known shortcomings [1]. Did you consider alternative or improved evaluation metrics (e.g., CMMD [1])?\n\n6. The substantial performance gap between SPTI and the PE image on the Cat dataset (Table 1) warrants further explanation.\n\n[1] Jayasumana, S., Ramalingam, S., Veit, A., Glasner, D., Chakrabarti, A., & Kumar, S. (2024). Rethinking fid: Towards a better evaluation metric for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9307-9315).\n\nLIMITATIONS:\nThe authors appropriately acknowledge that the captioning process may not preserve all image information and that domain-specific details could be lost. However, this limitation could be better quantified through targeted experiments, especially using downstream tasks that rely on high-fidelity semantics.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an interesting new idea to address the problem of differentially private synthetic image generation from text captions. Generating high quality DP images is hard since this leads to a poor privacy-utility tradeoff; on the other hand, DP text generation is relatively easy. This paper proposes a really nice idea of reducing private image generation to text-generation -- by first using a pre-trained model to caption an image, then a private image, then using a DP-finetuned model to produce a synthetic differentially private caption, and finally to use this caption and a pretrained text to image model to generate a similar synthetic image. \n\nThe main idea behind the paper is quite nice and the empirical results are quite strong.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n+ Elegant idea\n+ Strong empirical results that beats existing image baselines\n+ Well-written and clear presentation\n\nWeaknesses (mostly minor):\n\n- I know it is a bit difficult to fairly compare with image-only baselines, but some discussion on this would improve the paper further\n- The related work is missing references to other image-only methods, such as [1, 2, 3]\n[1] F. Harder, M. J. Asadabadi, D. J. Sutherland, and M. Park. Differentially private data generation\nneeds better features. arXiv preprint arXiv:2205.12900, 2022\n[2] https://arxiv.org/pdf/2309.00008\n[3] ] T. Cao, A. Bie, A. Vahdat, S. Fidler, and K. Kreis. Don’t generate me: Training differentially private generative models with sinkhorn divergence. Advances in Neural Information Processing Systems, 34:12480–12492, 2021\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nOne of the limitations of the paper is what happens when one of the pre-trained models are out-of-distribution wrt the private data. this could be discussed in more detail.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrivate Evolution is a promising approach to generating synthetic image and text data under differential privacy. This paper proposes a method, SPTI, which combines image and text private evolution methods. This method 1. maps private images to text using an off-the-shelf mutli-modal model, 2. uses an novel version of private evolution to obtain synthetic text, then 3. maps the synthetic text to images. The primary technical contribution is a version of private evolution that uses image voting, where both text and images are utilized to successively generate synthetic samples.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Interesting idea combining text and image approaches to private evolution. \n- Figure 1 is clear and helpful for understanding the proposed method. \n- The proposed method outperforms baseline methods with respect to FID score, and the image voting approach to private evolution outperforms using text alone. \n\nWeaknesses\n- There is not a formal proof in the main text or appendix that the proposed mechanism satisfies differential privacy. Moreover, the privacy analysis provided is insufficient. Most results such as the post-processing property are not clearly stated and are missing proper citations. Though stated on Line 188, the Advanced Composition Theorem (for Approx. DP) is not used for privacy accounting since the authors state that the privacy analysis uses RDP rather than Approx. DP.  \n- In the limitations section, the authors mention that the proposed method requires significantly more resources than baseline Private Evolution since the former requires several calls to muti-modal models; yet, there is no comparison of running time or token usage between the models.\n- There are several grammar mistakes throughout the text.\n\nQUESTIONS:\n1. In Table 1, where are the results for DP-finetune?\n2. Any thoughts on why Private Evolution works better on the Cat dataset than the proposed method? \n\nAdditional Comments\n- Rather than stating privacy results in terms of Approximate-DP, I suggest stating results in terms of RDP and provide the conversion from RDP to Approximate-DP. Moreover, it would be helpful to have a formal statement of the privacy result and a proof in either the main text or an appendix.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes synthesis via private textual intermediaries (SPTI) that is a framework for generating high-resolution differentially private synthetic images. The main idea is to shift the privacy-preserving synthesis process from the image domain to the text domain. The process begins by transforming private images into textual descriptions through an image captioning model. It then utilizes a modified Private Evolution algorithm to produce differential privacy (DP) text, followed by reconstructing images using a text-to-image diffusion model. Experiments on multiple datasets show that SPTI achieves better image quality (in terms of FID) than Private Evolution and DP fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1) The paper introduces a new framework, Synthesis via Private Textual Intermediaries (SPTI), which transitions the challenge of differentially private image synthesis from images to text. \n2) The method uses API-based models, including image captioning and text-to-image generation, without requiring model fine-tuning. \n3) The pipeline is modular and does not require DP fine-tuning of large image generation models, which reduces technical complexity compared to DP-SGD-based approaches, such as DP Fine-tuning.\n4) Experimental results show improvements over Private Evolution and DP fine-tuning, across multiple datasets and privacy budgets.\n\n**Weaknesses**\n1) The method has several modules: image-to-text, DP text evolution, and text-to-image synthesis. The interdependencies among these components make it challenging to assess their impact on overall performance.  In Section 4.3, the ablation study shows that the method is sensitive to changes in the image generation models. \n\n2) Using image-to-text as an intermediate representation has been explored in some existing works [1]. Although the application scenarios differ, the underlying idea is similar.\n \n      [1] Image2text2image: A novel framework for label-free evaluation of image-to-text generation with text-to-image diffusion models. \n\n3) It is unclear whether text-based representations are always sufficient to preserve fine-grained image details. \n\n4) The baseline comparisons are incomplete. While the primary comparisons focus on Private Evolution and DP fine-tuning, there is no evaluation against simpler baselines.\n   * Sampling random text prompts from the LLM API (without using any private data) to generate images, or generating images solely based on class names, raises uncertainty about whether this basic approach can achieve competitive FID scores.\n\n5) The DP guarantee is limited to the Private Evolution voting step, while other processes, particularly image-to-text captioning, access private data without DP protection. Although the paper cites the post-processing property of DP as justification, this design choice warrants further examination.\n\n6) Computational costs are not fully discussed. While the method avoids DP training, it requires repeated calls to large captioning models, LLMs, and text-to-image diffusion models. The paper reports that experiments were run on high-end GPUs, but no systematic analysis of compute costs compared to prior methods is provided. There are also additional costs for calling APIs such as ChatGPT.\n\n7) The paper does not fully analyze failure modes or limitations in the semantic fidelity of the generated images, especially when the text descriptions do not align well with the images in complex scene data.\n\nQUESTIONS:\n1) How sensitive is the overall performance to the choice of the image-to-text component? For example, if the captioning model were replaced with another model, such as LLaVA, how would this impact the image quality (FID)? Is there any quantitative analysis or ablation regarding this choice?\n\n2) For datasets with image tags or class labels, what would happen if the text input were simply 11an image of [tag]'' or a similar template for each image? Would this significantly affect performance or privacy guarantees?\n\n3) Are there any alternative evaluation methods, beyond FID scores, to assess the distribution alignment between the synthetic and private data? For example, can we use techniques like t-SNE or UMAP to visualize the embedding spaces of private and synthetic images in order to assess whether their distributions are well aligned? While the FID scores show improvements, it remains unclear whether the synthetic data truly captures the diversity and structure of the private data distribution.\n\n4) What is the estimated budget, in terms of computation and cost, for processing a datasets used in the paper, specifically for the image-to-text conversion using APIs like GPT or other captioning models?\n\nLIMITATIONS:\nNo. The paper recognizes the significant computational costs and API token usage but fails to provide specific figures or estimates. Without this information, it is challenging to evaluate the method's practical expense and determine if the cost is a reasonable trade-off compared to other DP approaches.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}