Title: Unlocking Markets: A Multilingual Benchmark to Cross-Market Question Answering
Benchmark: McMarket (specifically, the automatically translated Electronics category subset of a larger collected dataset for the MCPQA task)
Summary: The paper introduces Multilingual Cross-market Product-based Question Answering (MCPQA), a novel task where information from a resource-rich market (e.g., US) is used to answer product questions in a resource-scarce market, potentially in a different language. It presents a large-scale dataset derived from 17 Amazon marketplaces (11 languages), with a translated subset for Electronics called McMarket. Experiments on review-based answer generation (AG) and question ranking (QR) benchmark various models, demonstrating that leveraging cross-market information significantly boosts performance.
Contribution: Key contributions include: (1) Proposing the novel MCPQA task framework. (2) Constructing a large-scale, multilingual, cross-market PQA dataset, including the translated McMarket subset. (3) Demonstrating the use of LLMs (GPT-4) for annotating high-quality subsets (McMarket_r, McMarket_q) for specific tasks, validated by human assessment. (4) Providing extensive benchmarks comparing single-market vs. cross-market approaches using models from lexical methods to LLMs, verifying the benefit of cross-market data.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Product-related Question Answering (PQA), specifically focusing on cross-market information leveraging in a multilingual context.
Phenomenon defined by authors: Yes
Phenomenon definition: Multilingual Cross-market Product-based Question Answering (MCPQA) is defined as "providing answers to product-related questions in a main marketplace by utilizing information from another resource-rich auxiliary marketplace in a multilingual context". This involves using resources like reviews or QA pairs from an auxiliary market to address questions in a main market.
Definition scope: Subset
Definition integrity: Composite phenomenon
Phenomenon contested: Widely-agreed
Purpose notes: The work addresses the practical issue of data scarcity in smaller e-commerce marketplaces by proposing methods to leverage data from larger, resource-rich marketplaces, even across language barriers.
Task definition: The paper defines two subtasks within MCPQA: (1) Review-based Answer Generation (AG): Predict if a question is answerable using reviews from main and auxiliary markets, and if so, generate the answer. (2) Product-related Question Ranking (QR): Rank existing QA pairs from main and auxiliary markets based on their relevance for answering a given question in the main market.
Task item definition: The base dataset contains products with metadata, user questions, answers, and reviews from 17 Amazon marketplaces. The McMarket subset includes English translations. LLM-annotated subsets contain specific labels: McMarket_r has (Question, Reviews, Answerability, Generated Answer/Reason); McMarket_q has (Query Question, Candidate QA pair, Relevance Score, Reason).
Task definition detail: Key aspects are leveraging cross-market data (from a resource-rich auxiliary market like US) and handling multilingual information (via translation in McMarket).
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Product metadata and reviews originate from the XMarket dataset. Question-answer pairs were collected via web crawling from Amazon. Translations for McMarket were done using DeepL and NLLB models. Subsets McMarket_r and McMarket_q were annotated using GPT-4 prompts defined by the authors. Human validation of LLM annotations was performed by crowdworkers via Appen.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: The core task addresses answering real user questions on e-commerce platforms using available user-generated content like reviews and existing QAs.
Dataset size: McMarket (Electronics category subset): Over 2.2 million questions total. Test set sizes used in experiments: AG Test Set = 49,958; QR Test Set (McMarket_q) = 360.
Dataset size notes: AG Train/Validation sizes: 183,092 / 24,973. QR Train/Validation sizes (using McMarket_q): 1260 / 180.
Dataset metadata available: Yes
Dataset metadata detail: Includes marketplace origin, language, product identifiers/metadata, question text, answer text, review text, English translations (for McMarket), and LLM-generated annotations (answerability, generated answers, relevance scores, reasons) for the specific subsets. Timestamps are implicitly available based on analysis in Figure 3.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragarph)
Response format detail: Task AG involves generating natural language answers. Task QR involves producing a ranked list of relevant questions.
Metric definition: n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics), Distribution (perplexity, calibration, correlation)
Metric definition detail: AG: BLEU-4, ROUGE-L. QR: Mean Reciprocal Rank (MRR), Precision@3
Metric aggregation: Simple Mean
Metric statistics: BLEU-4, ROUGE-L, MRR, Precision@3. Mean scores are reported, sometimes with standard deviation (e.g., for text lengths in Table 2 ).
Metric subscores: Yes
Metric subscores detail: Results are reported per marketplace, for single-market vs. cross-market settings, and for translated vs. original language data in multilingual analysis. Performance is also compared between the main McMarket dataset and the LLM-annotated subsets.
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Cross-market data significantly increased the percentage of review-answerable questions across markets. Temporal analysis showed auxiliary market data often pre-dates main market questions. The human evaluation confirmed the high quality of GPT-4 annotations for AG (e.g., 88% correctness) and QR (97.6% F1), with LLM answers often preferred.
Taxonomy: Retrieval / 
New taxonomy: General Application / Document Retrieval
Procedural notes: Dataset built upon XMarket. Used DeepL and NLLB for translations. Used GPT-4 (gpt-4-1106-preview) for annotations, with prompts provided. Human validation via Appen. Data licensed under CCO 1.0 DEED for academic research. Baseline model details provided.
Other notes: The work highlights the utility of LLMs for dataset creation/annotation in specialized domains. It confirms the value of cross-context information transfer (cross-market, cross-product) for improving QA performance. Future work directions include improving multilingual handling without translation and exploring cross-lingual transfer techniques.