Package: docembedder
Patent breakthroughThe code in this repository is used to identify breakthrough innovations in historical patents from theUSPTO.
ThedocembeddingPython package contains a variety of methods for creating document embeddings. We have optimized and tested these methods for their ability to predict similarity between patents. This was done by maximizing the cosine similarity between patents that are classified into the same technology class, and minimizing cosine similarity between patents that fall into different technology classes. These methods with optimized parameters are then used to create document embeddings. From these embeddings, novelty scores are created using cosine similarities between the focal patent and patents in the previous n years and subsequent n years.Getting StartedClone this repository to your working station to obtain example notebooks and python scripts:git clone https://github.com/UtrechtUniversity/patent-breakthrough.gitPrerequisitesTo install and run this project you need to have the following prerequisites installed.- Python [>=3.8, <3.11]
- jupyterlab (or any other program to run jupyter notebooks)To install jupyterlab:pip install jupyterlabInstallationTo run the project, ensure to install the project's dependenciespipinstallgit+https://github.com/UtrechtUniversity/patent-breakthrough.gitBuilt withThese packages are automatically installed in the step above:scikit-learngensimsbertbpembUsage1. PreparationFirst you need to make sure that you have the data prepared. There should be a directory with *.xz files, which should have the year, so 1923.xz, 1924.xz, 1925.xz, etc. If this is not the case and you have only the raw .txt files, then you have to compress your data:fromdocembedder.preprocessor.parserimportcompress_rawcompress_raw(some_file_name,"year.csv",some_output_dir)Here, "year.csv" should be a file that that contains the patent ids and the year in which they were issued.2. Hyper parameter optimizationThere are procedures to optimize the preprocessor and ML models with respect to predicting CPC classifications. This is not a necessary step to compute the novelties and impacts, and has already been done for patents 1838-1951. For more information on how to optimize the models, see thedocumentation.3. PreprocessingTo improve the quality of the patents, and process/remove the start sections and such, it is necessary to preprocess these raw files. This is done using thePreprocessorandOldPreprocessorclasses, for example:fromdocembedder.preprocessorimportPreprocessor,OldPreprocessorprep=Preprocessor()old_prep=OldPreprocessor()documents=prep.preprocess_file("1928.xz")Normally however, we do not need to do preprocessing as a seperate step. We can compute the embeddings directly, which is explained in the next section.4. Embedding modelsThere are 5 different embedding models implemented to compute the embeddings:fromdocembedder.modelsimportCountVecEmbedder,D2VEmbedder,BPembEmbedderfromdocembedder.modelsimportTfidfEmbedder,BERTEmbeddermodel=BERTEmbedder()model.fit(documents)embeddings=model.transform(documents)These models can have different parameters for training, see the section on hyper parameter models. The result can be either sparse or dense matrices. The functions and methods in this package work with either in the same way.5. Computing embeddingsThe prepared data can be analysed to compute the embeddings for each of the patents using therun_modelsfunction. This function has the capability to run in parallel, in case you have more than one core on your CPU for examples.Before we can run, we have to tell docembedder the parameters of the run, which is done through theSimulationSpecificationclass:fromdocembedder.utilsimportSimulationSpecificationsim_spec=SimulationSpecification(year_start=1838,# Starting year for computing the embeddings.year_end=1951,# Last year for computing the embeddings.window_size=21,# Size of the window to compute the embeddings for.window_shift=1,# How many years between subsequent windows.debug_max_patents=100# For a trial run we sample the patents instead, remove for final run.)An example to create a file with the embeddings is:fromdocembedder.utilsimportrun_modelsrun_models({"bert":BERTEmbedder()},model,sim_spec,output_fp,cpc_fp)The output file is then a HDF5 file, which stores the embeddings for all patents in all windows.6. Computing novelty and impactTo compute the novelty and impact we're using theAnalysisclass:fromdocembedder.analysisimportDocAnalysiswithDataModel(output_fp,read_only=False)asdataanalysis=DocAnalysis(data)results=analysis.compute_impact_novelty("1920-1940","bert")The result is a dictionary that contains the novelties and impacts for each of the patents in that window (in this case 1920-1940).About the ProjectDate: February 2023Researcher(s):Benjamin Cornejo Costas (b.j.cornejocostas@uu.nl)Research Software Engineer(s):Raoul SchramShiva NadiMaarten SchermerCasper KaandorpJelle Treep (h.j.treep@uu.nl)LicenseThe code in this project is released underMIT license.Attribution and academic useManuscript in preparationContributingContributions are what make the open source community an amazing place to learn, inspire, and create. Any contributions you make aregreatly appreciated.To contribute:Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull RequestContactBenjamin Cornejo Costas -b.j.cornejocostas@uu.nlProject Link:https://github.com/UtrechtUniversity/patent-breakthrough