reviews: 
Comments: This paper proposes a method to optimize the answers provided by large language models by combining the properties of prompt tuning using an LLM and gradient based optimization. They compare their method against previous works that use either of these methods individually and showcase superior performance to the individual algorithms used in isolation.
Strengths: The paper is well written and presents the ideas in a clear manner. 

The idea itself is well motivated and should be easily replicatable.

One potentially really interesting part of this paper is the interpretability mentioned in line 464: This is indeed a unique advantage of the LLM based optimization above what can be provided by EA or Gradient based methods, and could potentially even be interesting to use as a standalone interpretation method (i.e. following the gradient, but keeping the result readable).
We encourage the authors to follow up on this.
Weaknesses: In the abstract you ask the question:
> “ Will this strategy also work well for solving complex non-convex optimization problems?”

This is not sufficiently answered (or arguably answered at all): This paper is not principally about optimization, but rather about prompt tuning. The title is also misleading: This is not a general optimization scheme that happens to have been studied in prompt tuning, but instead a technique _only_ applicable to prompt tuning. I would recommend the authors to remove the references towards more general optimization. Reading the title and abstract of this paper I would expect something closer to https://arxiv.org/pdf/2309.03409 than prompt tuning.

The “Algorithm 1” definition is quite messy: the embedding and token space projection operators are not inverses of each other, so shouldn’t be treated as such. E.g. in line 15 $\tilde p$ is reinitialized despite never being initialized in the first place.


Regarding token-space projection: In line 311 you claim that
> However, it is not feasible to directly provide the soft embedding vectors as input to the LLM,

This might be true with API models, but is not necessarily true for Llama models: As long as the tuned model and the model tuner are of the same type, one could directly input the continuous embeddings. This is done in textual inversion (https://arxiv.org/pdf/2208.01618), and is also the way that the original P-tuning (https://arxiv.org/pdf/2103.10385) worked. Are you also discretizing the P-tuning outputs?
As long as the prompt tuner and the prompt tunee is the same model, I see no reason why the inputs to the tuner have to be discretised first.
For gradient based tuners I would like to see a comparison on the same model with/without discretization since I suspect that the discretization is the true culprit for the performance drops, not the local optimality (indeed recent work, such as https://arxiv.org/pdf/2305.12827 suggests quasi-linearity of models when finetuning around pretrained models).

In line 515ff. you say that:
>A longer training process by gradient-based optimizer may not necessarily benefit the final prompt tuning results. We find that more iterations of gradient-based optimization may result in candidate prompts with less semantic diversity, which is not good for proposing LLM to generate more promising candidate prompts.

Doesn’t this show that the LLM can*not* escape local optima? If the LLM was necessary to escape the local optima, I would expect the LLM to make more of a difference the closer you are to the optimum.

In general, there are not enough comparisons against the state-of-the-art: from the LLM point of view I would expect a comparison against Zhou et al (https://arxiv.org/pdf/2211.01910) which also provide code to reproduce their experiments.
Due to the problem of gradient based optimization in discrete spaces, people have also started moving towards black-box combinatorial solvers, such as Guo et al’s Evolutionary method (https://arxiv.org/pdf/2309.08532, code is also available). Alternatively, you can also look at Deepmind’s PROMPTBREEDER (https://arxiv.org/pdf/2309.16797) which follows a similar idea
The reason I would like an ablation against the e.g. evolutionary method is because as-is it is unclear whether the LLM component of your framework is necessary or whether the advantage simply comes from the optimizer not being gradient based: “Is it necessary to have an LLM, or is it necessary to _not_ use gradients?”.

As a general note: when using APIs such as GPT-3.5 or GPT-4, please also give information of the time the experiments were done. APIs change all the time, so without knowing the timeframe the experiments were conducted in, it becomes hard to evaluate the results 2-3 years later.

In appendix A1 you list the prompt used for Llama2, which is quite different from the GPT-3.5/4 models: you mention that this is due to the inherent differences in the Llama and GPT models, but that opens up another question: How stable are the models to changes in prompts?
The prompt is a quite complex hyperparameter that you do not need in e.g. Evolutionary or gradient based methods, so one should compare different prompts as to not accidentally overfit the used prompt to the explored datasets.

Appendix A2 shows that the performance gap generally reduces as you increase the prompt length: Is there any interpretation for why this might be happening?
If I use 32 tokens, does this close the gap?


How do you set the learning rate for the gradient based systems? I've seen people use the traditional line-search with Wolfe/armijo rules to "get rid" of the learning rate hyperparameter.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3