Kati8/ICLR_2023_2025 huggingface dataset:
authors: Zixian Guo
authors: Ming Liu
authors: Zhilong Ji
authors: Jinfeng Bai
authors: Yiwen Guo
authors: Wangmeng Zuo
reviews: 
Comments: This paper proposes an optimization method that combines LLM and gradient descent to address the prompt tuning problem. The LLM generates potential prompts based on the historical path of gradient descent, and then these prompts are used as  restarting point for the next stage of gradient descent. This combined optimization approach ultimately achieves results that match or even surpass the current SOTA method, TCP, on several benchmark tasks.
Strengths: The paper innovatively combines the gradient-based optimizers and the optimization capabilities of LLMs, achieving good results on many benchmarks and providing detailed experimental details.
Weaknesses: The experiments are not very comprehensive. The paper only compares with two methods, CoOp and TCP (I am ignoring the results from Liu et al.), and it only compares with CoOp on Resnet50 and with TCP on ViT-B/16, lacking a comparison with the other method. Additionally, there is no comparison with other methods such as PromptSRC, MaPLe, and KgCoOp.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper presents an innovative approach to non-convex optimization by combining gradient-based methods with large language models (LLMs) to mimic a mentor-doer relationship. The gradient-based optimizer makes locally optimal search, while the LLM suggest possible improved starting points based on instructions that include the gradient-based optimizer's search history. The authors apply this approach to prompt tuning for both text models and vision-language models.
Strengths: 1. The paper introduces a novel combination of gradient-based methods and LLM-based planner. This approach is both creative and promising, as it leverages the complementary strengths of gradient-based optimization for local search and LLMs for strategic guidance.

2. The authors perform a comprehensive set of experiments to demonstrate the effectiveness and efficiency of their method. The breadth of experimentation, covering both text model and vision-language model prompt tuning, demonstrate the approach’s versatility and potential applicability across various domains.
Weaknesses: The main limitation of this approach is the unpredictable nature of the LLM-based planner. 
1. Based on results in Table 5, additional rounds of optimization do not always yield better outcomes, and N is too small to observe a clear performance trend in the optimization performance. 
2. Table 2  shows that smaller models like LLaMA-7B sometimes outperform larger models such as GPT-3.5 and even GPT-4. This suggests that the observed performance gains may not stem from any advanced reasoning ability inherent to the LLMs, but rather from task-specific nuances. 
3. Table 6 results show that increasing training iterations of the gradient optimizer leads to worse results, which suggests that the gradient-based optimizer and LLM planner may not be fully compatible, potentially causing interference rather than collaborative improvement.    

Overall, I personally believe the paper may overstate the readiness of LLMs as optimization planners, it may be premature to combine LLMs with gradient-based optimizer for complex optimization tasks.
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper introduces a collaborative optimization framework that combines gradient descent with LLMs to further enhance prompt tuning. In this method, gradient-based optimizer performs local updates while the LLM provides a new restarting point to help escape local optima. Experiments are conducted on language tasks and vision-language tasks.
Strengths: 1.	The proposed method is a hybrid method that combines the strengths of local gradient-based optimization and global LLM-based guidance.

2.	The experiments conducted in the paper are comprehensive.
Weaknesses: 1.	Based on the experiment results in both language and vision-language tasks, the improvement of the proposed method is mostly incremental, admittedly with a few cases where the improvement from the proposed method is significant. Additionally, as shown in Table 2, whether the proposed method is better than the baseline CoOp seems to depend on what LLM models are used too. 

2.	To better gauge the significance of the improvements, it would be beneficial for the authors to report standard deviations, especially given that some performance gains are as small as 0.07. This would provide a clearer picture of the method's reliability and consistency.

3.	Beyond accuracy metrics, I recommend that the authors also compare the running time and memory usage between the proposed method and baseline methods. This additional analysis would give valuable insights into the efficiency and practicality of the approach in real-world applications.
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3