## 1 Introduction  
Patents are a valuable source of information for understanding technological advancements, identifying emerging trends, and assessing the competitive landscape of various industries. However, because of the vast and growing volume of patent data and their unstructured nature, they are challenging to analyze manually [1]. Current approaches in patent analysis have primarily focused on manual interpretations, leading to challenges in scalability and consistency. This paper addresses that gap by evaluating the use of natural language processing (NLP) and machine learning (ML) techniques to extract insights from patent documents, particularly in the context of connected vehicle (CV) patents.  
Analysts have used NLP and ML to automate tasks such as patent classification, topic modeling, technology identification, and patent recommendation systems [2]. Even so, verifying the alignment of ML methods with human understanding of patent documents remains challenging and is an unresolved issue [3]. Hence, establishing a performance benchmark of state-of-the-art, free, open-source models will set the stage to evaluate the advancement of machines that can understand and classify patents. Such advancements could include large language models (LLMs), but accessing their application programminginterface (API) is not currently free or widely available to everyone, and the models are still immature.  
The goal of this research is to develop a comprehensive metric for quantifying alignment between subject matter expert (SME) and ML classification of patent topics. The author selected the field of CVs for the case study based on his more than 30 years of domain knowledge and industry experience inventing relevant products. With dozens of related patent awards in the field, the author amassed decades of experience reviewing and analyzing the nuances of patent documents in the CV and intelligent transportation systems domain. Hence, the author is qualified to serve as the SME in this study.  
The contribution of this paper is an original metric to quantify the alignment between SME classification and ML topic assignments. The proposed composite index comprises a purity factor that measures the average purity across SME classifications, and a dispersion factor that measures the spread of assigned topics across the SME classifications. The analysis of various mature methods provides valuable insights into the effectiveness of different combinations of tokenizers, normalizers, vectorizers, and topic modelers in the NLP pipeline. The embedded manifold visualizations effectively demonstrate the areas where the topic model aligns and diverges from the SME classifications, offering a clear view of the model's performance and limitations. This paper also bridges a gap in the literature between NLP efficiency and human-centered interpretability in topic modeling.  
The organization of the rest of this paper is as follows: Section 2 reviews the literature on CV technology development, the utility of NLP in patent analysis, and evaluations of NLP alignment with human judgment. Section 3 describes the data preparation, the NLP pipeline, and optimization of the selected topic model. Section 4 presents the results, including visualizations of the confusion between ML models and human interpretations. Section 5 discusses implications, further insights, and limitations of the work. Section 6 concludes the research and suggests future work.