Title: Intermediate Domain Alignment and Morphology Analogy for Patent-Product Image Retrieval
Venue: NeurIPS 2025
Accepted: yes
Keywords: open-set patent-product image retrieval

domain align

morphology analogy
Abstract: Recent advances in artificial intelligence have significantly impacted image retrieval tasks, yet Patent-Product Image Retrieval (PPIR) has received limited attention. PPIR, which retrieves patent images based on product images to identify potential infringements, presents unique challenges: (1) both product and patent images often contain numerous categories of artificial objects, but models pre-trained on standard datasets exhibit limited discriminative power to recognize some of those unseen objects; and (2) the significant domain gap between binary patent line drawings and colorful RGB product images further complicates similarity comparisons for product-patent pairs. To address these challenges, we formulate it as an open-set image retrieval task and introduce a comprehensive Patent-Product Image Retrieval Dataset (PPIRD) including a test set with 439 product-patent pairs, a retrieval pool of 727,921 patents, and an unlabeled pre-training set of 3,799,695 images. We further propose a novel Intermediate Domain Alignment and Morphology Analogy (IDAMA) strategy. IDAMA maps both image types to an intermediate sketch domain using edge detection to minimize the domain discrepancy, and employs a Morphology Analogy Filter to select discriminative patent images based on visual features via analogical reasoning. Extensive experiments on PPIRD demonstrate that IDAMA significantly outperforms baseline methods (+7.58 mAR) and offers valuable insights into domain mapping and representation learning for PPIR. (The PPIRD dataset is available at: \href{https://loslorien.github.io/idama-project/}{https://loslorien.github.io/idama-project/})


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new dataset and retrieval method for product-to-patent image retrieval. The proposed retrieval method first maps product image and patent image into the intermediate sketch domain, then replaces the mapped patent sketches with the most similar patent sketches through an morphology analogy filter, and finally extracts features for both product sketches and patent sketches. Evaluation on the introduced dataset demonstrates the superiority compared to a related method DeepPatent.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n1. This paper contributes a new large-scale dataset for product-patent retrieval, which would facilitate future studies on this task.\n2. This paper is well-written and easy to follow.\n3. Experiment results show apparent improvement over prior product-patent retrieval method.\n\n**Weaknesses**:\n1. Insufficient evaluation. The proposed method is only compared to DeepPatent. The task considered by this work belongs to cross-domain image retrieval, where there are many advanced methods that should be compared to verify the technical advantage of the proposed method, e.g., [1], [2].\n2. Novelty. Given that cross-domain image retrieval is widely studied, the novelty of product-patent retrieval needs justification.\n3. The design of intermediate domain. It is shown in Sec. 4.4 that mapping both product and patent image into edge map performs better than single domain adaptation, a typical way in cross-domain image retrieval. It is unclear why performing retrieval in an intermediate domain works better in original domains.\n\n[1] Li et al. Unsupervised Cross-Domain Image Retrieval via Prototypical Optimal Transport. AAAI 2024.\n\n[2] Hu et al. Unsupervised Feature Representation Learning for Domain-generalized Cross-domain Image Retrieval. ICCV 2023.\n\nQUESTIONS:\n1. How product-patent image retrieval distincts from existing cross-domain retrieval tasks, what are the unique challenges in this specific task?\n2. Is it possible to compare to existing cross-domain image retrieval methods?\n3. Why retrieval in sketch domain works better than color image domain and line-drawing domain? Is it sensitive to the choice of feature extractor?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the under-explored task of Patent-Product Image Retrieval (PPIR), where the goal is to retrieve relevant patent images based on input product images—useful for identifying potential IP infringements. The task is uniquely challenging due to: (a)A vast number of artificial object categories not present in standard vision datasets. (b) A significant domain gap between binary patent images and colorful product images.\nTo tackle these, the authors propose: (a) A new large-scale PPIR dataset (PPIRD), featuring over 700K patent images and nearly 4M unlabeled examples. (b) A novel method called IDAMA (Intermediate Domain Alignment and Morphology Analogy) - IDM maps both product and patent images to a shared sketch-like intermediate domain via edge detection, reducing domain discrepancy and MAF filters patent images using a class-agnostic confidence threshold (based on human-like morphological analogy), improving retrieval precision. (c) Empirical validation shows that IDAMA improves mean Average Recall (mAR) by up to +7.58 over baselines, and achieves strong performance with ViT backbones and contrastive pretraining.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The work targets a novel and impactful task with practical relevance to IP law, e-commerce, and legal AI.\n2. The use of a sketch-based intermediate domain is both intuitive and theoretically justified (via compressive sensing theory). Morphology Analogy Filter is inspired by cognitive science and adds interpretability to the filtering process.\n3. The creation of the PPIRD dataset is a significant contribution to the vision-for-patents community. The scale, realism (open-set formulation), and annotation process (with expert validation) are well executed.\n4. The paper benchmarks across multiple backbones (ResNet, ViT, Swin), pretraining schemes (supervised, MAE, IBOT), and domain alignment alternatives (e.g., colorization, binary line rendering). Ablation studies isolate the impact of IDM and MAF components.\n\nWeaknesses\n1. While the paper focuses on the patent-product scenario, it would benefit from a discussion on whether IDAMA generalizes to other domain-gap retrieval problems (e.g., art-photo, CAD-to-product).\n2. MAF relies on a threshold-based classifier score, which, although interpretable, is quite heuristic. A more principled or learnable mechanism might outperform it in general settings.\n3. The paper would be stronger with examples of failure cases—when does IDAMA fail and why?\n4. More visualizations of retrieval results (successes and failures) would enrich understanding, especially given the paper’s visually grounded motivation.\n\nQUESTIONS:\nPlease check weaknesses section\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the underexplored task of Patent-Product Image Retrieval (PPIR), which involves retrieving patent images based on product images to identify potential infringements. The key challenge of PPIR is bridging the domain gap between binary patent drawings and RGB product images while recognizing diverse unseen objects. To tackle these issues, the authors introduce an open-set image retrieval task, a comprehensive PPIR Dataset (PPIRD) and a method Intermediate Domain Alignment and Morphology Analogy (IDAMA). The PPIRD contains a test set with 439 product-patent pairs, a retrieval pool of 727,921 patents, and an unlabeled pre-training set containing 3,799,695 images. The experiments conducted on PPIRD demonstrate that IDAMA significantly surpasses baseline methods by +7.58 mAR.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.\tThis article introduces a new benchmark for Patent-Product Image Retrieval, comprising a test set of 439 product-patent pairs, a retrieval pool of 727,921 patents, and an unlabeled pre-training set featuring 3,799,695 images. This benchmark is likely to be significant for guiding future research directions in the field.\n2.\tThe motivation behind MAF is very interesting.\n3.\tThe effectiveness of IDM is analyzed through the theory of compressive sensing.\n\nWeakness:\n1.\tThe PPIR dataset is clearly a significant contribution of this paper, providing an essential resource for advancing research in patent-product image retrieval. However, the description of how this dataset was collected and curated is limited to section 4.1, which primarily focuses on the test set and retrieval pool construction. It leaves questions regarding the comprehensive collection process of all data, especially the unlabeled pre-training set consisting of 3,799,695 images. More detailed information on the methodologies and challenges involved in gathering such a vast amount of data would provide valuable insights and enhance the reproducibility of their work. \n2.\tI question the necessity of using real product images to search for patent drawings, as detailed textual descriptions or metadata often suffice for identifying potential infringements. The value of comparing cross-domain images (i.e., real product images and patent drawings) is debatable, given the increased system complexity, resource demands, and potential accuracy issues. Without clear evidence of significant advantages over existing methods, the practicality of such a specialized system remains uncertain. \n3.\tThe proposed IDAMA method appears overly simplistic and lacks significant innovation. The approach is commonly employed in cross-domain tasks. It seems that the authors have primarily adapted this existing strategy to a new application scenario rather than introducing a fundamentally novel contribution.\n\nQUESTIONS:\n1.\tHow did the authors collect such a large-scale dataset, and what are the sources of these data? Will the dataset be made publicly available in the future? Providing a detailed explanation of this process is crucial for ensuring the transparency and reproducibility of the research. Understanding the specific methods and challenges involved in data collection can assist other researchers in evaluating and replicating the experimental results.\n2.\tThe writing quality of this paper could be improved. Many sentences are overly complex and difficult to parse, which negatively affects the overall readability. For example, the following sentence is excessively long and lacks clarity: \"To address the challenge, we propose the Intermediate Domain Mapping (IDM) (See Fig. 2 for details) to align binary drawings patent images and colorful RGB product images by mapping both of them to the intermediate sketch domain.\" A more concise and clearly structured version would greatly enhance comprehension.\n3.\tThe proposed method lacks significant innovation. It primarily employs techniques that are commonly used in cross-domain tasks. Without introducing novel components, the contribution of this approach remains limited.\n4.\tI noticed that the comparative method, DeepPatent, uses mAP as its evaluation metric. Why did the authors choose to overlook this and rely solely on mAR for performance evaluation?\n\nLIMITATIONS:\nAlthough the paper highlights several technical challenges, such as the domain gap between product images and patent drawings, its experiments are limited to the authors' in-house PPIRD dataset. This restricts the evaluation across other data distributions, such as the DeepPatent dataset. Notably, although DeepPatent primarily consists of binary line drawings, it still offers valuable opportunities to assess the model's robustness to similar images of the same patent drawings. Expanding the experiments to more diverse datasets would provide a more comprehensive understanding of the proposed method's effectiveness and generalization capability.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}