Package: ptol
Pipleline to literatureA Pipeline for Obtaining Relevant Literature Based on Given KeywordsIt's a pipeline to help researchers accelerate literature searches and information acquisitionLet's start following the steps!Step 1Syntax for obtaining query syntaxes for databases such as PubMed based on keywordsCommon approachTake PubMed as an example.Take the subject keywords of our current study (e.g.Mycotoxin, enzyme, degrade, degradation, etc.) as an example.Website:https://pubmed.ncbi.nlm.nih.gov/advanced/Search based on search keyword statementsNote:When you use a literature database to search for relevant literature resources, we recommend that you optimize your keywords. For example, if your research area of interest is a physician topic, you should perform keyword validation at the MeSH URL (http://www.nlm.nih.gov/mesh/). This is to ensure that the most accurate research vocabulary is used. This maximizes the chance of ensuring that the literature resources searched in the database are the most accurate and relevant.Download all retrieved literature informationFor Web of Science:Website:https://www.webofscience.com/wos/woscc/advanced-searchYou can also supplement the relevant literature in other databases such as Google Scholar, Science Direct, etc.Common approachIn order to minimize manual operations, we hereby create a homemade tool,ptol, which can be downloaded and installed directly viapip install ptol. Once you have installed it, you can view the initial introduction to the subroutine with theptol -horptol --helpcommands. Below:The subroutinequery_syntaxin ptol automatically generates all possible lexical variations, PubMed and Web of Science query syntaxes, and corresponding download links based on keywords provided by the user.Module name:query_syntaxUsage:Enter the following command in the terminal to see help on using the program:ptolquery_syntax-hAll parameters and descriptions are listed below:ParametersDescriptions-mWhen running the script for the first time, use -m init to download the dictionary library first. Once downloaded, use -m run for subsequent run parameters.-iSetting the path to a file containing only keywords.-oSetting the output file path.Enter the file format:keyword 1keyword 2keyword 3...As shown in the figure below:Initialization for the first runptolquery_syntax-minit-ikeywords.txt-omy_result.txtWhen you see the last line of the terminal print "Successfully downloaded wordnet and other thesaurus!", indicating that the initialization is successful, then you can proceed to the use of the program.Practical training:ptolquery_syntax-mrun-ikeywords.txt-omy_result.txtOutputs the contents of the file:After that, according to the results given by this program, go directly from PubMed or Web of Science to download the results of searching literature information. You can refer to the next steps in the section1. common approach.Step 2Consolidation of literature informationLiterature collected from different databases was combined into one file through MS Excel. We keep only the Title and DOI number and save it as an xlsx file. Example:The file was then processed to remove duplicates using the subroutineremove_duplicates.Module name:remove_duplicatesUsage:Enter the following command in the terminal to see help on using the program:ptolremove_duplicates-hAll parameters and descriptions are listed below:ParametersDescriptions-iSetting the path to MS Excel files ending in .xlsx extension-oSetting the output file path.Practical training:ptolremove_duplicates-iall_database_literatures_data.xlsx-oall_database_literatures_data_single.txtOutputs the contents of the file:Step 3Download literaturesBased on the entirety of the relevant literature obtained earlier, a pdf of each piece of literature was downloaded.Note:In order to get all the above literature as fast as possible, we suggest that a one-time batch download can be realized by tools such asEndNote,crawler, and so on. Please note that at all times,please respect the copyrights of the authors and publishers of the literature. That is, the acquisition of the target literature is carried out through legal channels.Here, we provide the subroutinedownload_pdfsthat can batch download pdf format literature. Just for reference.Module name:download_pdfsUsage:Enter the following command in the terminal to see help on using the program:ptoldownload_pdfs-hNote:This subroutine is for test use by interested parties only, and in order to comply with the publisher's copyright, please download it from the official link of the literature publisher, or purchase the target literature you need.Step 4Convert pdf documents to text filesAfter downloading all the documents (pdf), use the subroutinepdf_to_textfor batch processing to convert all the documents into text files.Module name:pdf_to_textUsage:Enter the following command in the terminal to see help on using the program:ptolpdf_to_text-hAll parameters and descriptions are listed below:ParametersDescriptions-mThe script provides four kinds of pdf files into text files, respectively, numbered 1, 2, 3, 4, the user can set up according to their own preferences. A run, only one of the methods can be set. The purpose of such a design is that when some of the pdf documents can not be converted into text files, you can put these documents into a separate directory, try another method of conversion.-iSetting the path to the folder that includes only pdf-formatted literatures.-oSetting the path of output folder, all the text files which are converted successfully will be stored in this directory.Practical training:ptolpdf_to_text-m4-iliteratures_pdf-oliteratures_textView a text-formatted document from the leteratures_text folder as follows:Note:The file name of the document is logged in the terminal for failed conversions. Convenient for users to follow up.Access to large language modeling toolsAfter that, following the process described in our article, the research question is prepared manually and then the text file is copied and pasted into the input box of a big language model such asChatGPT. The goal of capturing information from the literature by big language models instead of manually can be realized.Finally, I sincerely hope that this pipeline can accelerate your research process and wish the best of luck in research.