Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to attribute generative models using digital watermarking. The paper explores the use of latent semantic dimensions as watermarks. Experiments are presented using StyleGAN2 and Latent diffusion models, which show that the approach is promising.The paper further showed that there is a tradeoff between generation quality and attribution accuracy. 
strength_and_weaknesses: The paper showed that there is a tradeoff between generation quality and attribution accuracy. 

Though some aspects of the paper are interesting, many parts are still not clear. Practical use case is missing. 

clarity,_quality,_novelty_and_reproducibility: Mostly clear. Not fully known how reproducible the paper is. 
summary_of_the_review: Though some aspects of the paper are interesting, many parts are still not clear. 

It is not clear how this approach will be useful in a practical scenario. Since watermarking is applicable in practical use cases, the authors should explain how the approach can be used practically. For example, are the authors suggesting to use a StyleGAN algorithm which also includes the deep watermarking method, and use this algorithm for generation?

It is not clear what the watermark actually is. Usually a watermark has some message or pattern embedded. 

Only StyleGAN2 and Latent diffusion models have been attributed. To be practically useful, more GANs or generative methods need to be experimented. 

In the post processing experiments, the post processing parameters are done for a fixed setting (example Guassian noise of -.1 standard deviation). It is also not clear what JPEG quality factor was used. Usually for post processing experiments, a range of values are considered and the analysis is then performed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method to produce watermarks on the content of generative models for the purpose of source tracking. The watermarks are added to the latent representations to produce outputs that are identifiable while preserving the quality of the distribution when compared to the original generative model. 


strength_and_weaknesses: Strengths: 
* The problem of source tracking the generated content of various generative models is very important.
* The proposed method is novel and conceptually interesting. 

Weakness: 

My main concern of this method is in the paper's evaluation setup and goals. In the paper, the authors only considered the following scenarios: (1) no post-process, (2) simple post-processing such as JPEG, noise blur etc.  Both (1) and (2) can be achieved easily by appending a blind image watermarking algorithm on top of the generated content. E.g. output = BlindWatermark(G(z), identifier). I believe the proposed baseline would actually outperform the suggested method on the simple distortions measured in this paper (See works such as [1] and [2]). 

I do believe there to be advantages in encoding in the latent semantic space that the baselines [1] and [2] cannot solve. Namely, if the "distortion" itself is a "deep learning based" attack. E.g., some conditional generative model which distorts the image far in the pixel space, but retains the semantic meaning.  But this is not measured at all in the paper and defeats the purpose of encoding information in the latent space. 

Other minor suggestions:
* More visual samples, and human rating results on the generated samples would help. FID alone isn't enough to determine visual sample quality. 
* The structure of the paper could be improved. Proposition statements should be moved to the appendix and Section 3 should contain clearly the step-by-step explanation of the watermark encoding and decoding algorithms. This would help the less experienced reader quickly understand the proposed method.
E.g. Encoding: Find d_phi principle component vectors of matrix H ....
Decoding: Solve Equation (Sec 4.3) to estimate the watermark signal phi. 


[1] Zhu, Jiren, et al. "Hidden: Hiding data with deep networks." Proceedings of the European conference on computer vision (ECCV). 2018.
[2] Luo, Xiyang, et al. "Distortion agnostic deep watermarking." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper could be structured better. It took me a while to sift through the actual method scattered throughout the proposition and experiment sections. 

Quality: See weakness.

Originality: Good.
summary_of_the_review: Overall I think the paper proposes an interesting idea, and experiment results show some promise of the proposed approach. But given the major concerns over the evaluation, I do not think the paper is ready for ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel approach for watermarking generative models (GANs + latent diffusion models). While previous approaches focused on adding watermarks in the image space, the paper's approach adds watermarks in the latent space to induce subtle semantic changes to the generated images. The latent directions of change are selected as the principal components but with small variances to avoid strong semantic changes. By using these latent directions to watermark the images, the procedure promises to be computationally efficient since it needs to only compute the directions once. The paper further provides a theoretical analysis of the accuracy-quality tradeoff and empirically evaluates the approach on StyleGAN2 and Latent Diffusion models.
strength_and_weaknesses: Strengths:
+ Tackles an important problem of generative models: watermarking the generative models (and their generated images) to allow attribution of contents to their source models.
+ Most parts of the paper are clearly written and easy to follow. Particularly the introduction section states a good entry into the problem.
+ The proposed solution of adding watermarks in the latent space instead of the image space offers an interesting avenue.
+ The empirical results promise that the model indeed beats previous approaches in terms of accuracy-quality tradeoff and induces less noticeable changes to the images.
+ The formal analysis of the accuracy-quality tradeoff provides interesting insights into the problem.

Weaknesses:
- Parts of section 3 are partially imprecise and difficult to read. Rewriting some parts of it and adding some intuition would make following it easier. Also, a list of symbols in the appendix might be worth considering.
- I expect the approach to only work for generative models with disentangled latent spaces. So for other types of GANs without using, e.g., a mapping network to entangle the dimensions, the approach is not applicable. Similarly, I am not sure if the approach works (in principle) with other diffusion models, e.g., imagen or DALL-E 2.
- I also miss some information on the computation needed to perform the attribution step. Since it needs to solve an optimization that involves forward passes through the generator, it might take some time. So I think approaches with watermarking in the image space need to train separate attribution networks, which is costly at training but fast at inference. And the proposed approach is fast at watermarking (does not need to train a separate attribution model) but the attribution process might be slow / computation costly. If this is true, the trade-off should at least be stated in the paper.
- Unfortunately, no source code is provided to reproduce the results
clarity,_quality,_novelty_and_reproducibility: Whereas some aspects and the intuition of the approach are clear, some other aspects, particularly Section 3, are hard to follow. For example, the sentence "Let l : Rdx × Rdx → R be a distance metric between two images, (α̂, ϕ̂) the estimates." in 3.2 does not make clear, how the estimates parameters are related to the distance metrics. Readability can be improved by providing more details in the whole section and improving sentence structures.

Besides, the quality of the writing and experiments is good. However, some abbreviations, such as SG2, should be introduced. 

Overall, the approach is novel and adds an interesting avenue to the problem of watermarking. I am not aware of previous research proposing a similar approach but I am also not deep into the watermarking literature.

I expect most parts of the approach could be implemented following the description in the paper. However, since no hyperparameters and seed are stated and no source code is provided, I expect an exact reproduction of the results to be impossible.

Smaller remarks:
- The figures, particularly in the appendix, should be added as pdf files to enable loss-less zooming. 
- The statement at the end of section 4, "Attacks with large pixel-wise perturbations can be perceptually insignificant." should be supported by some reference.
- The URL in the reference of Kelly is not fitting the page margin.
summary_of_the_review: In summary, I like the proposed idea and direction of the paper, as well as the empirical evaluation, and think that it will benefit the community if the paper would get accepted. It tackles an important problem of generative models that becomes even more important with increasing model quality and recent advances in text-to-image synthesis. However, the quality of the writing should be improved, and details for reproduction should be added, in the best case together with the source code.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new method to improve watermarks for generative models, particularly investigating the trade-off between attribution accuracy and generation quality. The proposed method explores latent semantic space as watermark, by mean of principal components and eigenvectors. The paper also explores how the design parameters of the method can be used to establish a practical and computationally inexpensive trade-off. Experiments show that the method outperforms the SOTA that uses shallow watermarking.
strength_and_weaknesses: 
Strengths:

1) the paper is well written and well organised, presenting convincing evidence that it outperforms the SOTA
2) the theoretical derivations are thorough and seem also convincing

Weaknesses:
1) this authors only compare their method with the method of Kim et al. (2020). The method should be compared against more methods that explore different properties of watermarking. For instance:

Matthew Tancik, Ben Mildenhall, and Ren Ng. Stegastamp: Invisible hyperlinks in physical photographs.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 2117–2126, 2020. [exploring printer-proof watermarking]

Shadmand, Farhad, Iurii Medvedev, and Nuno Gonc̗alves. "CodeFace: A Deep Learning Printer-Proof Steganography for Face Portraits." IEEE Access 9 (2021): 167282-167291. [exploring printer-proof watermarking]

Ruowei Wang, Chenguo Lin, Qijun Zhao, and Feiyu Zhu. Watermark faker: towards forgery of
digital image watermarking. In 2021 IEEE International Conference on Multimedia and Expo
(ICME), pp. 1–6. IEEE, 2021. [exploring resistance to watermark attack]

clarity,_quality,_novelty_and_reproducibility: The paper is clear and show goos quality.
It present a novel method.
As the code is not available, it is difficult to reproduce.

summary_of_the_review: The paper is interesting and presents a convincing method for watermarking. However, the paper would benefit of comparing its results against more difficult scenarios (printer-camera) and alternative methods.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper investigates the use of latent semantic dimensions as watermarks for attributing generative models. It conducts theoretical analysis of the accuracy-quality tradeoff and evaluates the proposed method on both StyleGAN2 and latent diffusion models.

Strengths:
- It studies an important problem 
- A novel watermarking approach
- Sound derivation of the method and theoretical analysis of the accuracy-quality tradeoff

Weaknesses:
- Writing quality can be further improved beside the clarification in the rebuttal
- Reviewers have concern on over-claims that external deep learning watermarking modules can be pruned trivially. It's not a sufficient argument for not comparing with those baselines. Additional comments from a reviewer after reading rebuttal:
Given more sophisticated methods (such as hiding the watermarking module with other non-watermarking layer), it is not obvious to me why it would be trivial to prune the watermarking network. The author has not provided more details on how to do this either. Therefore, I also disagree with the assessment that it is an "unfair" comparison with external watermarking modules. Further more, if pruning is considered (i.e., in a white-box setting), the authors should also evaluate watermarking removal strategies for their proposed method as well, which was not done in this paper.
- More choices of attacks should be included in evaluation.
Comments from a reviewer, "I think with carefully selected post-processing, perhaps even just horizontal flipping, watermark detection could be bypassed"

While the clarification and experiments from the rebuttal are useful to improve the clarity and addressed some concerns, this paper would require more revision to improve the writing quality and addition comparison with more baselines in order to meet the standard at ICLR.
justification_for_why_not_higher_score: The main concern on the lack of comparison with existing deep learning based watermarking methods is not well addressed. The authors' rebuttal fail to convince the reviewer it is not necessary to make that comparison. Also, multiple reviewers agree the more thorough post-process attacks should be included in the evaluation to test the limitation of the method.
Decision: Reject