IntentionQA: A Benchmark for Evaluating Purchase Intention Comprehension Abilities of Large Language Models in E-commerce
Authors: Wenxuan Ding, Weiqi Wang, Sze Heng Douglas Kwok, Minghao LIU, Tianqing Fang, Jiaxin Bai, Xin Liu, Changlong Yu, Zheng Li, Chen Luo, Bing Yin, Junxian He, Yangqiu Song


Reviewer paper summaries: This paper provides a dataset for evaluating e-commerce users' intentions in buying a product and likely next product to purchase as two separate subtasks. The dataset is multiple choice and each task has easy/medium/hard examples to use.
The dataset is automatically created by leveraging existing resources and is then spot checked by human evaluators (MTurk) to ascertain how well the automatic creation worked.
Multiple LLM-based systems are then evaluated.

Note for the authors: I would remove the LLM part of the title since the dataset is more broadly applicable than that. There is nothing constraining its use to LLM-based systems.
Summary of strengths: - Two related benchmarking datasets provided for a key e-commerce issue of determining and then using user purchase intent
- Evaluation of several systems broken down by system type
- Good error analysis
Summary of weaknesses: - The human annotation correctness (Table 2) shows that for task 1 "hard", the correctness is only 85% which is too low for a benchmark dataset. Since there are only 118 of these (table 1), they should all be manually corrected. Task 2 "medium" is also low for a benchmark at 92.6%.
- As mentioned in the summary, this benchmark is more broadly applicable than LLMs and this should be reflected in a title and intro change (at which point this becomes a strength).
Comments, suggestions and typos: Suggestions in order of the paper:

Provide information on the license for the dataset.

Intention needs to be defined early on.  It can be defined by an example (and if Fig 1 is modified as below, that could just be referenced to save space).

Fig 1: Instead of this figure, it would be more helpful to have an actual example, perhaps from the appendix.  The little images are hard to see, but the detailed product descriptions as in the appendix make the actual task and data much clearer.

Sec 3.2: It would be helpful to have a short example of what is an ok intention and what is overly general.

Sec 3.4: It is notoriously hard to threshold on cosine similarity. Given the broad ranges here, it is probably ok, especially since some human correlation was done, but a mention of the issues would be good.

Sec 4.2: Great to see the human annotation. Please provide IAA for the multi-judgement results. Also, given the 89% correct, can something be said about the 11% errors?

l410: Still => Again (or Similarly)

Table 2: The human annotation correctness (Table 2) shows that for task 1 "hard", the correctness is only 85% which is too low for a benchmark dataset. Since there are only 118 of these (table 1), they should all be manually corrected. Task 2 "medium" is also low for a benchmark at 92.6%.  Also, the average is artificially high for task 1 since there are relatively few hard examples.
Update 4.2.2 to reflect these issues.

Table 32: Why is task 2 "medium" so much harder for humans than easy and hard?
Indicate whether top system results are stat sig compared to the next best.

l530: For the 8.3% implausible, are these wildly so and if so, any idea why they are chosen? There is probably not room, but providing an example would be good.

Limitations/Ethics: Perhaps mention that there may be culturalization that is not captured by this (e.g. are these US or EU based) and that minimally this is English only and so a good initial dataset but should later be extended.
Overall assessment scores: 3