as a prompt engineer your task is a design step by step prompt from this instruction ,please notify that you should use {placeholders} for each step according my research title and any output of any step,{لطفا در هر مرحله از این روش ،پرامپتی ک تعریف میکنی ورودی و خروجی مورد انتظارشم بنویس و همینطور علاوه بر متنی ک تولید میکنه ب عنوان خروجی یک متنی هم استاندارد ب عنوان ورودی مراحل دیگر تولید کند }
A APPENDIX
A.1 EVALUATION METRICS
Table 5: Evaluation metrics of ideas.
Metric Definition
Novelty Are the problems or approaches new? Is this a novel combination of
familiar techniques? Is it clear how this work differs from previous contributions? Is related work adequately referenced?
Significance Are the idea important? Are other people (practitioners or researchers)
likely to use these ideas or build on them? Does the idea address a difficult problem in a better way than previous research? Does it provide a
unique theoretical or pragmatic approach?
Clarity Is the paper clearly written? Is it well-organized? Does it adequately
inform the reader?
Feasibility Can the idea be realized with existing technology or methods? Are there
any technical difficulties or bottlenecks? Is the idea clear and logical?
Is there any obvious error or unreasonable part in the idea, and can the
experiment be designed normally according to this idea.
Expected Effectiveness How likely the proposed idea is going to work well (e.g., better than
existing baselines).
Table 6: Evaluation metrics of experiment design.
Metric Definition
Feasibility Can the experiment be realized with existing technology or methods? Are there any
technical difficulties or bottlenecks? Is the experimental plan detailed and feasible?
Are the experimental steps clear and logical? Is there any obvious error or unreasonable part in the experiment. Consider the rationality of its steps and the possibility that
the idea can be successfully implemented.
Quality Is there a clear rationale for each step of the experimental design? Are the baseline
and evaluation metrics chosen appropriately? Has the design taken into account the
potential advantages and limitations of the methods used? Can this experimental design effectively support the claims made in the idea.
Clarity Is the experimental plan clearly written? Dose it provide enough information for the
expert reader to understand the experiment? Is it well organized? Does it adequately
inform the reader?
A.2 SPECIFIC PROMPTS
Here are the prompts used in this paper.
• Prompts used in CoI construction
– Prompt used to convert a topic into a search query for literature retrieval (Table 7)
– Prompt used to evaluate whether a paper is relevant to the topic (Table 8)
– Prompt used to extract idea, experiment, entities and references from paper (Table 9)
and 10
– Prompt used to summarize current trends of CoI (Table 11)
• Prompts used in idea generation
14
Preprint
– Prompt used to predict future trend (Table 12 and 13)
– Prompt used to generate idea (Table 14 and 15)
– Prompt used to check the novelty of the idea (Table 16)
• Prompts used in experiment design
– Prompt used to generate experiment design (Table 17)
– Prompt used to review experiment design (Table 18)
– Prompt used to get queries for search paper to refine experiment design (Table 19)
– Prompt used to refine experiment (Table 20)
• Prompts used in benchmark construction
– Prompt used to extract topic from real paper (Table 21)
– Prompt used to extract the idea from real paper (Table 22)
– Prompt used to extract the experiment design from real paper (Table 23)
• Prompts used in idea arena
– Prompt used to compare two ideas (Table 24)
– Prompt used to compare two experiment designs (Table 25)
Table 7: Prompt used to convert a topic into a search query for literature retrieval
You are a master of literature searching, tasked with finding
relevant research literature based on a specific topic.
Currently, we would like to study the following topic: [Topic]
Please provide the literature search queries you would use to
search for papers related to the topic and idea.
Each query should be a string and should be enclosed in double
quotes. It is best to output one query representing the whole and
other queries representing different aspects of the whole.
Output strictly in the following format:
Queries: ...
A.3 ADDITIONAL EXPERIMENT RESULTS
We present the evaluation results of idea generation for both model-based evaluation (including
GPT-4o, Gemini-1.5-Pro-Exp-0827, and Claude-3.5-Sonnet) and human-based evaluation in Table
26.
15
Preprint
Table 8: Prompt used to evaluate whether a paper is relevant to the topic
You are an expert researcher tasked with evaluating whether a given
paper is relevant to our research topic based on its title and
abstract.
Below are the details of the paper you need to assess:
Title: [Title]
Abstract: [Abstract]
The topic is: [Topic]
If the paper title and abstract are related to the topic, output
1; otherwise, output 0. As long as you feel that this article has
reference value for your question, you can use it to help you study
the topic, it does not need to be completely consistent in topic.
Please follow the strict format below:
Think: ...
Relevant: 0/1
Table 9: Prompt used to extract idea, experiment, entities and references from paper (part I)
You are a scientific research expert, tasked with extracting and
summarizing information from provided paper content relevant to
the topic: [Topic]. Your deliverables will include pertinent
references, extracted entities, a detailed summary, and the
experimental design.
The topic you are studying is: [Topic] (Ensure that the references
are pertinent to this topic.)
Extraction Requirements:
Entities:
1. Identify unique entities mentioned in the paper, such as model
names, datasets, metrics, and specialized terminology.
2. Format the entities with a name followed by a brief
description.
3. Ensure all entities are relevant to the specified topic
([Topic]).
Summary Idea:
1. Background: Elaborate on the task’s context and previous work,
outlining the starting point of this paper.
2. Novelty: Describe the main innovations and contributions of
this paper in comparison to prior work.
3. Contribution: Explain the primary methods used, detailing the
theory and functions of each core component.
4. Detail Reason: Provide a thorough explanation of why the
chosen methods are effective, including implementation details for
further research.
5. Limitation: Discuss current shortcomings of the approach.
Continue to next table →
16
Preprint
Table 10: Prompt used to extract idea, experiment, entities and references from paper (part II)
Experimental Content:
1. Experimental Process: Detail the entire experimental
procedure, from dataset construction to specific steps, ensuring
clarity and thoroughness.
2. Technical Details: Describe any specific technologies
involved, providing detailed implementation processes.
3. Clarity of Plan: State your experimental plan concisely to
facilitate understanding without unnecessary complexity.
4. Baseline: Elaborate on the baseline used, comparative methods,
and experimental design, illustrating how these support and
validate the conclusions drawn.
5. Verification: Explain how your experimental design assists in
verifying the core idea and ensure it is detailed and feasible.
Relevance Criteria:
1. Method Relevance: References must directly correlate with the
paper’s methodology, indicating improvements or modifications.
2. Task Relevance: References should address the same task, even
if methods differ, better have the same topic [Topic]
3. Baseline Relevance: References should serve as baselines for
the methods discussed in the paper.
4. Output Format: Provide references without author names or
publication years, formatted as titles only.
The paper content is as follows: [Paper content]
Please provide the entities, summary idea, experimental design,
and the three most relevant references (Sort by relevance, with
priority given to new ones with the same level of relevance, do not
reference the original paper.) based on the paper’s content.
Note: Ensure the references are pertinent to the topic you are
studying: [Topic]. If there are no relevant references, output
[].
Now please output strictly in the following format:
Entities: ...
Idea: ...
Experiment: ...
References: ...
17
Preprint
Table 11: Prompt used to get trends of CoI
You are a scientific research expert tasked with summarizing the
historical progression of research related to our current topic,
based on the literature we have reviewed.
Here are the entities you need to know : [Entities]
The topic you are studying is: : [Topic]
The literature from early to late: [Idea chain]
Your objective is to outline the historical evolution of the
research in light of current trends. Please follow these
requirements:
Analysis of Published Viewpoints: Examine the progression of ideas
across the identified papers. Detail how each paper transitions
to the next--for instance, how Paper 0 leads to Paper 1, and so
forth. Focus on understanding how Paper 1 builds upon the concepts
in Paper 0. Elaborate on specific advancements made, including
proposed modules, their designs, and the rationale behind their
effectiveness in addressing previous challenges. Apply this
analytical approach to each paper in the sequence.
Please present your findings in the following format:
Trends:
Paper 0 to Paper 1: ...
Paper 1 to Paper 2: ...
...
Table 12: Prompt used to predict future trend (Part I)
You are a scientific expert tasked with formulating a novel and
innovative research idea based on your comprehensive literature
review. Your objective is to propose a feasible approach that
could significantly advance the field.
Here are the entities you need to know : [Entities]
The literature you have studied is as follows: [Chain of ideas]
The following section delineates the progressive relationships
among the previously summarized research papers: [Trend]
Based on previous research, analyze how human experts think
and transition from previous methods to subsequent approaches.
Focus on their reasoning logic and the sources of their thought
processes. Learn to emulate their reasoning patterns to further
develop and guide your own research direction in a natural and
coherent manner.
Additionally, you are encouraged to adopt the following three modes
of thinking:
Continue to next table →
18
Preprint
Table 13: Prompt used to predict future trend (Part II)
1. Reflection: Reflect on scenarios where a specific method
encounters significant challenges. Consider potential solutions
that could effectively address these issues, make the solutions
sounds reasonable, novel and amazing.
2. Analogy: Identify a specific problem you are currently
facing and research existing solutions that have successfully
tackled similar challenges. Explore these solutions and adapt
key principles and strategies to your situation. Think creatively
about how tools and approaches from other domains can be
re-imagined to devise a novel strategy for your issue. Encourage
you to actively explore methods in other fields to solve your
current problems.
3. Deep Dive: Some methods may present specific approaches to
addressing a particular problem. Consider whether there are
aspects that could be modified to enhance their rationale and
effectiveness.
Note:Each article’s limitations are specific to that particular
piece and should not be applied to others. Carefully consider the
task at hand and analyze the potential issues you might encounter
if you proceed with your original approach, reflecting on the
challenges previously faced. Then, think critically about how to
address these issues effectively.
You are encouraged to apply human reasoning strategies to identify
future research directions based on prior studies. Aim for
in-depth analysis rather than mere integration of existing ideas.
Please avoid introducing unfamiliar information, ensuring that
the trends you present are both authentic and reasonable. Before
proposing any trends, take a moment to reflect on the principles
underlying the methods you’re employing and assess their relevance
to your research area.
The future research direction should be related to the topic:
[Topic]
Please present the future research direction in the following
format:
Future direction: ...
19
Preprint
Table 14: Prompt used to generate idea (part I)
You are a scientific expert tasked with formulating a novel and
innovative research idea based on your comprehensive literature
review. Your objective is to propose a feasible approach that
could significantly advance the field.
The following are examples of ideas you have proposed in the past
that are similar to real papers. Please avoid this situation as
much as possible. You can continue to make in-depth innovations,
but avoid plagiarism: [Bad case]
Here are the entities you need to know: [Entities]
The topic you are studying is: [Topic]
The literature you have studied is as follows: [Chain of ideas]
Your idea is composed of the following components:
Motivation:
1. Provide a background for your idea, summarizing relevant work.
2. Identify shortcomings in previous research and highlight the
specific problems that remain unsolved and that you aim to address.
Novelty:
1. Distinguish your proposed method from existing methods
(preferably by naming specific approaches).
2. Detail the improvements of your method compared to past work.
3. Clearly outline at least three contributions your idea offers
to the field, including the problems it resolves and the benefits
it delivers.
Method:
1. Present a detailed description of your idea, focusing on the
core method, the specific problem it solves, and enhancements over
earlier research (citing relevant literature with titles).
2. Explain the step-by-step methodology, including the functions
of each module and the rationale for why this approach effectively
addresses previous challenges.
Please adhere to the following guidelines:
1. Your research idea should be innovative, feasible, and
contribute meaningfully to the field. Please carefully examine
the idea you have proposed, avoid immediate perception, and try to
be different from the previous methods as much as possible.
2. Ensure your proposal is solid, clearly defined, and practical
to implement. Logic should underpin your reasoning.
3. Write in clear, concise language aimed at an audience with
limited background knowledge in the subject. Avoid complex
technical jargon, but when professional terms are necessary,
provide thorough explanations.
4. Refrain from introducing concepts from uncertain fields to
prevent proposing ideas that may be incorrect or impractical.
5. When referencing other research, please include the titles of
the cited papers.
6. Please avoid introducing unfamiliar information, ensuring that
the trends you present are both authentic and reasonable. Before
proposing any trends, take a moment to reflect on the principles
underlying the methods you’re employing and assess their relevance
to your research area.
Continue to next table →
20
Preprint
Table 15: Prompt used to generate idea (part II)
7. Each article’s limitations are specific to that particular
piece and should not be applied to others. Carefully consider the
task at hand and analyze the potential issues you might encounter
if you proceed with your original approach, reflecting on the
challenges previously faced. Then, think critically about how to
address these issues effectively.
The following section delineates the progressive relationships
among the previously summarized research papers: [Trend]
The following section outlines the potential future research
directions based on the literature you have studied: [Future
direction]
Please output your motivation,novelty,method firstly and then
output your final idea.The final idea should clearly explain the
origins, motivation, and challenges of your idea, detailing how you
overcame these hurdles.
Please present the final idea in the following format:
Motivation: ...
Novelty: ...
Method: ...
Final idea: ...
Table 16: Prompt used to check the novelty of the idea
You are a scientific research expert tasked with evaluating the
similarity between a specified idea and existing research. Your
objective is to determine if the target idea closely resembles any
findings in the provided papers.
The target idea you need to check is as follows: [Idea]
The relevant papers you need to refer to are as follows:[Content of
retrieved papers]
Here are your guidelines:
1. Comparison Process: Begin by thoroughly comparing each
paper’s ideas with the target idea. Consider the methodologies,
conclusions, and underlying concepts in each paper in your
analysis.
2. Similarity Assessment: If the target idea shares fundamental
similarities with any existing research to the extent that they can
be considered identical, classify this as plagiarism.
3. Output: Your output should provide a clear thought process,
the similarity assessment, a summary of the target idea, and the ID
of the most relevant similar paper.
Please output strictly in the following format:
Think: ...
Similar: 0/1
Summary of the idea: ...
Similar paper id: 0 to n
21
Preprint
Table 17: Prompt used to generate experiment
You are a scientific expert tasked with designing rigorous,
feasible experiments based on specified scientific questions
and the methodologies derived from the idea I provide, along
with relevant past research. Your goal is to assist researchers
in systematically testing hypotheses and validating innovative
discoveries that could significantly advance their fields.
Past Related Research Experiments: [Past experiments]
Here are the entities you need to know: [Entities]
Here is the idea you need to design an experiment for: [Idea]
Please propose a detailed experimental plan addressing the
following points:
1. Experimental Design: Develop rigorous experiments to
ensure the reliability and validity of your results. Provide
a comprehensive explanation of the baseline used, comparative
methods, ablation study design, and criteria for data analysis
and result evaluation. Clarify how these components collectively
reinforce and validate the conclusions of your research. Structure
your experimental design in a clear, logical, and step-by-step
manner, ensuring each step is well-defined and easy to understand.
2. Implementation of Technologies/Methods: If your experimental
design involves specific technologies or methodologies, describe
the implementation process in detail, including key technical
aspects. For any critical concepts utilized, provide thorough
explanations. For instance, if you propose a modular approach,
detail its construction, components, and functionality.
3. Feasibility Assessment: Ensure your experimental plan is
realistic, considering technological availability, timelines,
resources, and personnel. Identify potential challenges and
propose strategies for addressing them.
4. References to Previous Studies: When citing related
literature, include titles and pertinent details of the original
papers. Strive to use as many references as necessary to support
your experimental design.
5. Visual Aids: If useful, provide pseudo code or a flowchart to
illustrate the implementation process. For example, you can use
pseudo code to detail the core algorithm or the model architecture,
or employ a flowchart to map out the experimental procedure and
data flow.
6. Clarity of Language: Use straightforward language to describe
your methods, assuming the reader may have limited knowledge of
the subject matter. Avoid complex jargon and utilize accessible
terminology. If professional terms are necessary, please provide
clear and detailed explanations.
Please output strictly in the following format:
Experiment:
Step1: ...
Step2: ...
...
22
Preprint
Table 18: Prompt used to review experiment
You are an expert in paper review. Your task is to analyze whether
a given experiment can effectively verify a specific idea, as well
as assess the detail and feasibility of the experiment.
Here are the related entities you need to know: [Entities]
The idea presented is: [Idea]
The corresponding experiment designed for this idea is:
[Experiment]
Please conduct your analysis based on the following criteria:
1. Can the experiment validate the idea? If not, identify
the issues and suggest improvements to enhance its verification
capability and feasibility.
2. Are there specific experimental procedures that are confusing
or poorly designed? Discuss any methods that may not be feasible,
uncertainties in constructing the dataset, or a lack of explanation
regarding the implementation of certain methods.
3. Evaluate the clarity, detail, reasonableness, and feasibility
of the experimental design.
4. Provide suggestions for improving the experiment based on the
shortcomings identified in your analysis.
5. Focus solely on the experiment design; please refrain from
altering the original idea.
6. Ensure that your suggestions are constructive, concise, and
specific.
Please strictly follow the following format for output:
Suggestion: ...
Table 19: Prompt used to get query for search paper to refine experiment
You are a research expert tasked with refining and improving an
experimental plan based on the feedback received.
The experimental plan you proposed is as follows: [Experiment]
You have received the following suggestions for improvement:
[Suggestions]
Please decide whether you need to search for relevant papers to
obtain relevant knowledge to improve your experiment.
If you need to search for relevant papers, please provide a search
query for literature search, else provide "".
For example: if suggestions say that the dynamic query additional
information and update knowledge graph described in the experiment
is not clearly described, so you need to output "dynamic knowledge
graph update".
Please output strictly in the following format:
Query:...
23
Preprint
Table 20: Prompt used to refine experiment
You are a research expert tasked with refining and improving an
experimental plan based on the feedback received.
The information of the literature you maybe need to refer to are
as follows: [Searched paper information]
The experimental plan you proposed is as follows: [Experiment]
Please propose a detailed experimental plan addressing the
following points:
1. Experimental Design: Develop rigorous experiments to
ensure the reliability and validity of your results. Provide
a comprehensive explanation of the baseline used, comparative
methods, ablation study design, and criteria for data analysis
and result evaluation. Clarify how these components collectively
reinforce and validate the conclusions of your research. Structure
your experimental design in a clear, logical, and step-by-step
manner, ensuring each step is well-defined and easy to understand.
2. Implementation of Technologies/Methods: If your experimental
design involves specific technologies or methodologies, describe
the implementation process in detail, including key technical
aspects. For any critical concepts utilized, provide thorough
explanations. For instance, if you propose a modular approach,
detail its construction, components, and functionality.
3. Feasibility Assessment: Ensure your experimental plan is
realistic, considering technological availability, timelines,
resources, and personnel. Identify potential challenges and
propose strategies for addressing them.
4. References to Previous Studies: When citing related
literature, include titles and pertinent details of the original
papers. Strive to use as many references as necessary to support
your experimental design.
5. Visual Aids: If useful, provide pseudo code or a flowchart to
illustrate the implementation process. For example, you can use
pseudo code to detail the core algorithm or the model architecture,
or employ a flowchart to map out the experimental procedure and
data flow.
6. Clarity of Language: Use straightforward language to describe
your methods, assuming the reader may have limited knowledge of
the subject matter. Avoid complex jargon and utilize accessible
terminology. If professional terms are necessary, please provide
clear and detailed explanations.
You have received the following suggestions for
improvement:[Suggestions]
Please refine your experimental plan based on the feedback
provided. Ensure your refined plan is feasible, clearly defined,
and addresses the feedback you received.
Please output strictly in the following format:
Experiment: ...
24
Preprint
Table 21: Prompt used to extract topic from real paper
You are a research expert tasked with extracting the main topic
from the provided paper information.
The main topic should encompass broad fields such as "Retrieve
augment generation" or "using diffusion models for video
generation". However, it should also include a relevant task to
the topic, formatted as "topic:... task:...".
Please read the provided paper and extract only the topic, which
should follow this structure.
The paper’s title is [Title]
The paper’s abstract is as follows: [Abstract]
The paper’s introduction is as follows: [Introduction]
Please output strictly in the following format:
topic: ...
25
Preprint
Table 22: Prompt used to extract idea from real paper
You are a research expert tasked with extracting the main idea from
the provided paper information.
The main idea should encompass the motivation, solved problem,
novelty, method of the paper.
Please read the provided paper and extract the main idea from the
paper.
The paper content is as follows: [Content]
Idea is composed of the following components:
Motivation: Explain the background of the idea and past related
work, identify the shortcomings of past work, identify the problems
that need improvement, and identify the issues the paper want to
address.
Novelty: Explain the differences between the method and the
current method (preferably list specific methods), explain what
improvements the paper have made to the previous method, and then
identify the problems that can be solved and the benefits that can
be gained from these improvements.
Method: Provide a detailed description of your idea, including the
core method, the problem it solves, and the improvement compared
with previous work(Cite the previous work with the title of the
paper). Explain the specific steps of the method, the specific
functions of each module, and the specific reasons why this method
can solve the previous problem.
Here are some tips for extracting the main idea:
1. Make idea easy to understand, use clear and concise language
to describe, assuming the reader is someone who has few knowledge
of the subject, avoid using complex technical terms, and try to
use easy-to-understand terms to explain.If the paper use some
professional terms, please explain them in detail.
2. When the paper cite other papers, please indicate the title of
the original paper.
The final idea should be detailed and specific, clearly explain
the origins, motivation, novelty, challenge, solved problem and
method of the paper, and detail how the overcame these hurdles.
Ensure your approach is innovative, specifying how this innovation
is reflected in your experimental design.
The final idea should be double-blind, i.e. no experimental
results or codes should be shown.
Please output strictly in the following format:
Final idea: ...
26
Preprint
Table 23: Prompt used to extract experiment from real paper
You are a research expert tasked with extracting the specific
experiment steps from the provided paper information.
The specific experiment steps should include the specific methods
for each step.
Please read the provided paper and extract specific experiment
steps from the paper.
The paper content is as follows: [Content]
There are some tips for extracting the experiment steps:
1. Detail the Experimental Process: Describe the entire
experimental process, including how to construct the dataset and
each specific experimental step. Ensure that each experimental
method is clearly and thoroughly detailed.
2. If specific technologies are involved in the experimental
design, describe the implementation process in as much detail as
possible (i.e., technical details)
3. Make sure your experimental plan is concise and clear, and can
be easily understood by others,should not be too complicated.
4. Please provide a detailed explanation of the baseline used in
the paper, the comparative methods, the ablation design and the
experimental design. Specifically, elaborate on how these elements
collectively support and validate the conclusions drawn in your
research.
5. Explain how your experimental design can help you verify the
idea and how the experiment is detailed and feasible.
Now please output strictly in the following format:
Experiment:
Step1: ...
Step2: ...
...
27
Preprint
Table 24: Prompt used to compare two ideas
You are a judge in a competition. You have to decide which idea is
better.
The idea0 is: [idea0]
The idea1 is: [idea1]
The topic is: [topic]
Which idea do you think is better? Please write a short paragraph
to explain your choice.
Here are your evaluation criteria:
1. Novelty: Are the problems or approaches new? Is this a novel
combination of familiar techniques? Is it clear how this work
differs from previous contributions? Is related work adequately
referenced?
2. Significance: Are the idea important? Are other people
(practitioners or researchers) likely to use these ideas or build
on them? Does the idea address a difficult problem in a better way
than previous research? Does it provide a unique theoretical or
pragmatic approach?
3. Feasibility: Can the idea be realized with existing technology
or methods? Are there any technical difficulties or bottlenecks?
Is the idea clear and logical? Is there any obvious error or
unreasonable part in the idea, and can the experiment be designed
normally according to this idea.
4. Clarity: Is the paper clearly written? Is it well-organized?
Does it adequately inform the reader?
5. Effectiveness: How likely the proposed idea is going to work
well (e.g., better than existing baselines).
Note:
Avoid any position biases and ensure that the order in which
the responses were presented does not influence your decision.
DO NOT allow the LENGTH of the responses to influence your
evaluation, choose the one that is straight-to-the-point instead
of unnecessarily verbose. Be as objective as possible. (very
important!!!)
If you think idea0 is better than idea1, you should output 0. If
you think idea1 is better than idea0, you should output 1. If you
think idea0 and idea1 are equally good, you should output 2.
Your output should be strictly in following format:
Your thinking process: ...
Your choice:
Novelty: 0/1/2
Significance: 0/1/2
Feasibility: 0/1/2
Clarity: 0/1/2
Effectiveness: 0/1/2
28
Preprint
Table 25: Prompt used to compare two experiments
You are a judge in a competition. You have to decide which
experiment is better.
The idea of experiment0 is: [idea0]
The experiment0 is: [experiment0]
The idea of experiment1 is: [idea1]
The experiment1 is: [experiment1]
Which experiment do you think is better? Please write a short
paragraph to explain your choice.
Here are your evaluation criteria:
1. Feasibility: Can the experiment be realized with existing
technology or methods? Are there any technical difficulties or
bottlenecks? Is the experimental plan detailed and feasible? Are
the experimental steps clear and logical? Is there any obvious
error or unreasonable part in the experiment. Consider the
rationality of its steps and the possibility that the idea can be
successfully implemented.
2. Quality: Is there a clear rationale for each step of the
experimental design? Are the baseline and evaluation metrics
chosen appropriately? Has the design taken into account the
potential advantages and limitations of the methods used? Can
this experimental design effectively support the claims made in the
idea.
3. Clarity: Is the experimental plan clearly written? Dose it
provide enough information for the expert reader to understand the
experiment? Is it well organized? Does it adequately inform the
reader?
Note: Avoid any position biases and ensure that the order in which
the responses were presented does not influence your decision.
DO NOT allow the LENGTH of the responses to influence your
evaluation, choose the one that is straight-to-the-point instead
of unnecessarily verbose. Be as objective as possible. (very
important!!!)
If you think experiment0 is better than experiment1, you should
output 0. If you think experiment1 is better than experiment0,
you should output 1. If you think experiment0 and experiment1 are
equally good, you should output 2.
Your output should be strictly in following format:
Your thinking process: ...
Your choice:
Feasibility: 0/1/2
Quality: 0/1/2
Clarity: 0/1/2