Package: triplea
triple-aArticle Analysis AssistantThis program somehow creates a network of article references and provides a connection between authors and keywords, these things are usually called "Citation Graph".There are various software and online systems for this, a brief review of which can be foundhere.This tool gives you the power to create a graph of articles and analyze it. This tool is designed as aCLI(command-line interface) and you can use it as a Python library.🎮 Main FeaturesHow to useSetupFunctional UseTraining NER for Article TitleCommand Line (CLI) UseGet and Save list of article identifier base on search termMove core pipeline stateRun custom pipelineNER Article TitleCountry-based Co-authorshipExtract Triple from AbstractExtract Topic from AbstractImport Single Reference FileExport graphVisualizing GraphAnalysis GraphWork with Article RepositoryConfigurationTestingDependenciesUse caseBio BankRegistry of Breast CancerEHRGraph VisualizationGraph AnalysisKnowledge ExtractionRelated ArticleCode QualityCitationLicense🎮 Main FeaturesSingle paper analysisDynamic citations deep definition for meta data fetchNetwork Analysis (Per Node/Overall Graph)Import bibliography fileUse forBibliometric AnalysisHow to useFrom SourceSetupClone repository:gitclonehttps://github.com/EhsanBitaraf/triple-a.gitorgitclonegit@github.com:EhsanBitaraf/triple-a.gitCreate environment variable:python-mvenvvenvActivate environment variable:Windows$.\venv\Scripts\activateLinux$sourcevenv/bin/activateInstall poetry:pipinstallpoetryInstal dependences:poetryinstallrun cli:poetryrunpythontriplea/cli/aaa.pyFunctional Useget list of PMID in state 0term='("Electronic Health Records"[Mesh]) AND ("National"[Title/Abstract]) AND Iran'get_article_list_all_store_to_kg_rep(term)move from state 1move_state_forward(1)get list of PMID in state 0 and save to file for debugginf usedata=get_article_list_from_pubmed(1,10,'("Electronic Health Records"[Mesh]) AND ("National"[Title/Abstract])')data=get_article_list_from_pubmed(1,10,'"Electronic Health Records"')data1=json.dumps(data,indent=4)withopen("sample1.json","w")asoutfile:outfile.write(data1)open before file for debugging usef=open('sample1.json')data=json.load(f)f.close()get one article from kg and save to filedata=get_article_by_pmid('32434767')data=json.dumps(data,indent=4)withopen("one-article.json","w")asoutfile:outfile.write(data)Save Title for Annotationfile=open("article-title.txt","w",encoding="utf-8")la=get_article_by_state(2)forainla:try:article=Article(**a.copy())except:passfile.write(article.Title+"\n")Training NER for Article TitleYou can use NLP(Natural Language Processing) methods to extract information from the structure of the article and add it to your graph. For example, you can extract NER(Named-entity recognition) words from the title of the article and add to the graph.Here's how to create a custom NER.Command Line (CLI) UseBy using the following command, you can see the command completionhelp. Each command has a separatehelp.python.\triplea\cli\aaa.py--helpoutput:Get and Save list of article identifier base on search termGet list of article identifier like PMID base on search term and save into knowledge repository in first state (0):use this command:python.\triplea\cli\aaa.pysearch--searchterm[searchterm]Even the PMID itself can be used in the search term.python.\triplea\cli\aaa.pysearch--searchterm36467335output:Move core pipeline stateThe preparation of the article for extracting the graph has different steps that are placed in a pipeline. Each step is identified by a number in the state value. The following table describes the state number:List of state numberStateShort DescriptionDescription0article identifier savedAt this stage, the article object stored in the data bank has only one identifier, such as the PMID or DOI identifier1article details article info saved (json Form)Metadata related to the article is stored in theOriginalArticlefield from theSourceBank, but it has not been parsed yet2parse details infoThe contents of theOriginalArticlefield are parsed and placed in the fields of the Article object.3Get Citation4Get Full TextAt this stage, the articles that are open access and it is possible to get their full text are taken and added to the bank5Convert full text to string-1Errorif error happend in move state 1 to 2-2Errorif error happend in move state 2 to 3There are two ways to run a pipeline. In the first method, we give the number of the existing state and all the articles in this state move forward one state.
In another method, we give the final state number and each article under that state starts to move until it reaches the final state number that we specified.
The first can be executed with thenextcommand and the second with thegocommand.With this command move from current state to the next statepython.\triplea\cli\aaa.pynext--state[currentstate]for example move all article in state 0 to 1:python.\triplea\cli\aaa.pynext--state0output:gocommand:python.\triplea\cli\aaa.pygo--end[laststate]python.\triplea\cli\aaa.pygo--end3output:Run custom pipelineApart from the core pipelines that should be used to prepare articles, customized pipelines can also be used. Custom pipelines may be implemented to extract knowledge from texts and NLP processing. These pipelines themselves can form a new graph other than the citation graph or in combination with it.List of Custom PipelineActionTag NameDescriptionPrerequisiteTriple extraction from article abstractFlagExtractKGAt least core state 2Topic extraction from article abstractFlagExtractTopicAt least core state 2Convert Affiliation text to structural dataFlagAffiliationMiningThis is simple way for parse Affiliation textAt least core state 2Convert Affiliation text to structural dataFlagAffiliationMining_TitipatauseTitipat Achakulvisut Repofor parsing Affiliation textAt least core state 2Text embedding abstract and send to SciGeniusFlagEmbeddingAt least core state 2Title and Abstract Review by LLMFlagShortReviewByLLMAt least core state 2NER Article TitleYou can try the NER method to extract the major topic of the article's title by using the following command. This command is independent and is used for testing and is not stored in the Arepo.python.\triplea\cli\ner.py--title"The Iranian Integrated Care Electronic Health Record."Country-based Co-authorshipA country-based co-authorship network refers to a network of collaborative relationships between researchers from different countries who have co-authored academic papers together. It represents the connections and collaborations that exist among researchers across national boundaries.By studying a country-based co-authorship network, researchers can gain insights into international collaborations, identify emerging research trends, foster interdisciplinary cooperation, and facilitate policy decisions related to research funding, academic mobility, and scientific development at a global scale.There are several software tools available that can help you produce country-based co-authorship networks. Here are a few popular options:VOSviewer: VOSviewer is a widely used software tool for constructing and visualizing co-authorship networks. It offers various clustering and visualization techniques and allows you to analyze and explore the network based on different attributes, including country affiliation.Sci2 Tool: The Science of Science (Sci2) Tool is a Java-based software package (inGitHub) that supports the analysis and visualization of scientific networks. It offers a variety of functionalities for constructing and analyzing co-authorship networks, including country-based analysis. It allows users to perform data preprocessing, network analysis, and visualization within a single integrated environment.To convert affiliation into a hierarchical structure of country, city and centers, you can use the following command:python.\triplea\cli\aaa.pypipeline-nFlagAffiliationMiningExtract Triple from Abstractpython.\triplea\cli\aaa.pypipeline--nameFlagExtractKGExtract Topic from Abstractpython.\triplea\cli\aaa.pypipeline--nameFlagExtractTopicAn example of working with the functions of this part usingJupyteris given inhere. which is finally drawn using VOSviewer program as below:Import DataImport Single Reference FileImport file type is.bib,.enw,.rispython.\triplea\cli\importbib.py"C:\...\bc.ris"output:Import Triplea Formatpython.\triplea\cli\aaa.pyimport--helppython.\triplea\cli\aaa.pyimport--typetriplea--formatjson--barTrue"C:\BibliometricAnalysis.json"Export DataVarious data export can be created from the article repository. These outputs are used to create raw datasets.TypeFormattripleajson, csv ,csvsrayyancsvRefMan*risIt has not yet been implemented.For guidance from the export command, you can act like this:python.\triplea\cli\aaa.pyexport--helpFor Example :The export is limited to 100 samples, and the resulting exported articles are saved in the file Triple Json format named "test_export.json".python.\triplea\cli\aaa.pyexport--typetriplea--formatjson--limit100--output"test_export.json"python.\triplea\cli\aaa.pyexport--typetriplea--formatjson--output"test_export.json"Export Triplea CSV format:python.\triplea\cli\aaa.pyexport--typetriplea--formatcsv--output"test_export.csv"python.\triplea\cli\aaa.pyexport--typetriplea--formatcsvs--output"export.csv"Export for Rayyan CSV format:python.\triplea\cli\aaa.pyexport--typerayyan--formatcsv--output"test_export.csv"Export Graphfor details information:python.\triplea\cli\aaa.pyexport_graph--helpMaking a graph with thegraphmlformat and saving it in a filetest.graphmlpython.\triplea\cli\aaa.pyexport_graph-ggen-all-fgraphml-o.\triplea\testMaking a graph with thegexfformat and saving it in a fileC:\Users\Dr bitaraf\Documents\graph\article.gexf.This graph contains article, author, affiliation and relation between them:python.\triplea\cli\aaa.pyexport_graph-garticle-author-affiliation-fgexf-o"C:\Users\Dr bitaraf\Documents\graph\article"Making a graph with thegraphdictformat and saving it in a fileC:\Users\Dr bitaraf\Documents\graph\article.json.This graph contains article, Reference, article cite and relation between them:python.\triplea\cli\aaa.pyexport_graph-garticle-reference-garticle-cited-fgraphdict-o"C:\Users\Dr bitaraf\Documents\graph\article.json"Making a graph with thegraphmlformat and saving it in a fileC:\graph-repo\country-authorship.jgraphmlson.This graph contains article, country, and relation between them:python.\triplea\cli\aaa.pyexport_graph-gcountry-authorship-fgraphml-o"C:\graph-repo\country-authorship"Types of graph generators that can be used in the-gparameter:NameDescriptionstoreIt considers all the nodes and edges that are stored in the databasegen-allIt considers all possible nodes and edgesarticle-topicIt considers article and topic as nodes and edges between themarticle-author-affiliationIt considers article, author and affiliation as nodes and edges between themarticle-keywordIt considers article and keyword as nodes and edges between themarticle-referenceIt considers article and reference as nodes and edges between themarticle-citedIt considers article and cited as nodes and edges between themcountry-authorshipTypes of graph file format that can be used in the-fparameter:NameDescriptiongraphdictThis format is a customized format for citation graphs in the form of a Python dictionary.graphjsongsongpickleWrite graph in Python pickle format. Pickles are a serialized byte stream of a Python objectgraphmlThe GraphML file format uses .graphml extension and is XML structured. It supports attributes for nodes and edges, hierarchical graphs and benefits from a flexible architecture.gexfGEXF (Graph Exchange XML Format) is an XML-based file format for storing a single undirected or directed graph.Visualizing GraphSeveral visualizator are used to display graphs in this program. These include:Alchemy.js: Alchemy.js is a graph drawing application built almost entirely in d3.interactivegaraph: InteractiveGraph provides a web-based interactive visualization and analysis framework for large graph data, which may come from a GSON filenetwulf: Interactive visualization of networks based on Ulf Aslak's d3 web app.python.\triplea\cli\aaa.pyvisualize-garticle-reference-garticle-cited-p8001python.\triplea\cli\aaa.pyvisualize-ggen-all-p8001output:python.\triplea\cli\aaa.pyvisualize-garticle-topic-garticle-keyword-p8001output:Visulaize FileA file related to the extracted graph can be visualized in different formats with the following command:python.\triplea\cli\aaa.pyvisualize_file--formatgraphdict"graph.json"Analysis Graphanalysis infocommand calculates specific metrics for the entire graph. These metrics include the following:Graph Type:SCC:WCC:Reciprocity :Graph Nodes:Graph Edges:Graph Average Degree :Graph Density :Graph Transitivity :Graph max path length :Graph Average Clustering Coefficient :Graph Degree Assortativity Coefficient :python .\triplea\cli\aaa.py analysis -g gen-all -c infooutput:Creates a graph with all possible nodes and edges and calculates and lists the sorteddegree centralityfor each node.python .\triplea\cli\aaa.py analysis -g gen-all -c sdcoutput:Work with Article RepositoryArticle Repository (Arepo) is a database that stores the information of articles and graphs. Different databases can be used. We have used the following information banks here:TinyDB- TinyDB is a lightweight document oriented databaseMongoDB- MongoDB is a source-available cross-platform document-oriented database programTo get general information about the articles, nodes and egdes in the database, use the following command.python.\triplea\cli\aaa.pyarepo-cinfooutput:Numberofarticleinarticlerepositoryis1220Node(s)inarticlerepository.0Edge(s)inarticlerepository.122article(s)instate3.Get article data by PMIDpython.\triplea\cli\aaa.pyarepo-pmid31398071output:Title   : Association between MRI background parenchymal enhancement and lymphovascular invasion and estrogen receptor status in invasive breast cancer.
Journal : The British journal of radiology
DOI     : 10.1259/bjr.20190417
PMID    : 31398071
PMC     : PMC6849688
State   : 3
Authors : Jun Li, Yin Mo, Bo He, Qian Gao, Chunyan Luo, Chao Peng, Wei Zhao, Yun Ma, Ying Yang, 
Keywords: Adult, Aged, Breast Neoplasms, Female, Humans, Lymphatic Metastasis, Magnetic Resonance Imaging, Menopause, Middle Aged, Neoplasm Invasiveness, Receptors, Estrogen, Retrospective Studies, Young Adult,Get article data by PMID and save toarticle.jsonfile.python.\triplea\cli\aaa.pyarepo-pmid31398071-oarticle.jsonanother command fo this:python.\triplea\cli\aaa.pyexport_article--idtypepmid--id31398071--formatjson--output"article.json"ConfigurationFor details information:python.\triplea\cli\aaa.pyconfig--helpGet environment variable:python.\triplea\cli\aaa.pyconfig-cinfoSet new environment variable:python.\triplea\cli\aaa.pyconfig-cupdateBelow is a summary of important environment variables in this project:Environment VariablesDescriptionDefault ValueTRIPLEA_DB_TYPEThe type of database to be used in the project. The database layer is separate and you can use different databases, currently it supportsMongoDBandTinyDBdatabases. TinyDB can be used for small scope and Mango can be used for large scopeTinyDBAAA_TINYDB_FILENAMEFile name of TinyDBarticledata.jsonAAA_MONGODB_CONNECTION_URLStandard Connection String FormatFor MongoDBmongodb://user:pass@127.0.0.1:27017/AAA_MONGODB_DB_NAMEName of MongoDB CollectionarticledataAAA_TPS_LIMITTransaction Per Second Limitation1AAA_PROXY_HTTPAn HTTP proxy is a server that acts as an intermediary between a client and PubMed server. When a client sends a request to a server through an HTTP proxy, the proxy intercepts the request and forwards it to the server on behalf of the client. Similarly, when the server responds, the proxy intercepts the response and forwards it back to the client.AAA_PROXY_HTTPSHTTPS ProxyAAA_REFF_CRAWLER_DEEP1AAA_CITED_CRAWLER_DEEP1From PackageYou can create a python virtual environment before installing and it is recommended that you do so.$python-mvenvvenv$.\venv\Scripts\activateInstall Package with pip:$pipinstalltripleaCreate environment variable by.envfile:TRIPLEA_DB_TYPE = TinyDB
AAA_TINYDB_FILENAME = articledata.json
AAA_MONGODB_CONNECTION_URL = mongodb://localhost:27017/
AAA_MONGODB_DB_NAME = articledata
AAA_TPS_LIMIT = 1
AAA_PROXY_HTTP = 
AAA_PROXY_HTTPS = 
AAA_REFF_CRAWLER_DEEP = 1
AAA_CITED_CRAWLER_DEEP = 1
AAA_TOPIC_EXTRACT_ENDPOINT=http://localhost:8001/api/v1/topic/
AAA_CLIENT_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0"If you do not create the mentioned file, the default values will be used, which are:TRIPLEA_DB_TYPE = TinyDB
AAA_TINYDB_FILENAME = default-tiny-db.json
AAA_TPS_LIMIT = 1
AAA_REFF_CRAWLER_DEEP = 1
AAA_CITED_CRAWLER_DEEP = 1
AAA_TOPIC_EXTRACT_ENDPOINT=http://localhost:8001/api/v1/topic/
AAA_CLIENT_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0"Run CLI:$aaa--helpoutput:Usage:aaa[OPTIONS]COMMAND[ARGS]...

Options:-v,--version--helpShowthismessageandexit.

Commands:analysisAnalysisGraph.configConfigurationadditionalsetting.exportExportarticlerepositoryinspecificformat.export_articleExportArticlebyidentifier.export_graphExportGraph.export_llmExportpreTrainLLM.goMovesthearticlesstateintheArepountilendstate.importimportarticlefromspecificfileformattoarticle...importbibimportarticlefrom.bib,.enw,.risfileformat.nerSingleNERwithcustommodel.nextMovesthearticlesstateintheArepofromthecurrent...pipelineRunCustomPipeLineinarepo.searchSearchqueryfromPubMedandstoretoArepo.Note: The visualization function is only available in the source versionTestingpoetryrunpytestpoetry run pytest tests/poetryrunpytest--covFor unit test check :bibilometric:3728301835970485DependenciesFor graph analysis:networkxFor NLP:PyTextRanktransformersspaCyFor data storage:TinyDBpy2neopymongoFor visualization of networks:netwulfAlchemy.jsInteractiveGraphFor CLI:clickFor packaging and dependency management:PoetryUse caseWith this tool, you can create datasets in different formats, here are examples of these datasets.Breast CancerPubmed Query:"breast neoplasms"[MeSH Terms] OR ("breast"[All Fields] AND "neoplasms"[All Fields]) OR "breast neoplasms"[All Fields] OR ("breast"[All Fields] AND "cancer"[All Fields]) OR "breast cancer"[All Fields]495,012resultsConfiguration:AAA_MONGODB_DB_NAME = bcarticledata
AAA_REFF_CRAWLER_DEEP = 0
AAA_CITED_CRAWLER_DEEP = 0EDirectused.Search with this command:python .\triplea\cli\aaa.py search --searchterm r'"breast neoplasms"[MeSH Terms] OR ("breast"[All Fields] AND "neoplasms"[All Fields]) OR "breast neoplasms"[All Fields] OR ("breast"[All Fields] AND "cancer"[All Fields]) OR "breast cancer"[All Fields]'if --searchterm argument is too complex use this:python .\triplea\cli\aaa.py searchby Filter :{
    "mindate" : "2022/01/01",
    "maxdate" : "2022/12/30"
}Get info of all downloaded article:python.\triplea\cli\aaa.pyarepo-cinfooutput:Numberofarticleinarticlerepositoryis309140Node(s)inarticlerepository.0Edge(s)inarticlerepository.30914article(s)instate0.Run Core pipeline to next statuspython.\triplea\cli\aaa.pynext--state0then parsing article:python.\triplea\cli\aaa.pynext--state1Extract Triple is type of custom pipeline. you can run this:python.\triplea\cli\aaa.pypipeline--nameFlagExtractKGBio BankPubmed Query:"Biological Specimen Banks"[Mesh] OR BioBanking OR biobank OR dataBank OR "Bio Banking" OR "bio bank"39,023resultsSearch with this command:python.\triplea\cli\aaa.pysearch--searchterm"\"Biological Specimen Banks\"[Mesh] OR BioBanking OR biobank OR dataBank OR \"Bio Banking\" OR \"bio bank\" "Get 39,023 result until2023/01/02"ERROR":"Search Backend failed: Exception:\n\'retstart\' cannot be larger than 9998. For PubMed, ESearch can only retrieve the first 9,999 records matching the query. To obtain more than 9,999 PubMed records, consider using EDirect that contains additional logic to batch PubMed search results automatically so that an arbitrary number can be retrieved. For details see https://www.ncbi.nlm.nih.gov/books/NBK25499/"This query had more than 10,000 results, and as a result, the following text was used:To retrieve more than 10,000 UIDs from databases other than PubMed, submit multiple esearch requests while incrementing the value of retstart (see Application 3). For PubMed, ESearch can only retrieve the first 10,000 records matching the query. To obtain more than 10,000 PubMed records, consider using<EDirect>that contains additional logic to batch PubMed search results automatically so that an arbitrary number can be retrieved.This is hard code inget_article_list_from_pubmedmethods inPARAMS.This Query Added lately:"bio-banking"[Title/Abstract] OR "bio-bank"[Title/Abstract] OR "data-bank"[Title/Abstract]9,012resultspython.\triplea\cli\aaa.pysearch--searchterm" \"bio-banking\"[Title/Abstract] OR \"bio-bank\"[Title/Abstract] OR \"data-bank\"[Title/Abstract] "after run this. get info:Number of article in article repository is 47735Exportgraphmlformat:python.\triplea\cli\aaa.pyexport_graph-garticle-reference-garticle-keyword-fgraphml-o.\triplea\datasets\biobank.graphmlRegistry of Breast CancerKeyword Checking:"Breast Neoplasms"[Mesh]  
"Breast Cancer"[Title]
"Breast Neoplasms"[Title]  
"Breast Neoplasms"[Other Term]
"Breast Cancer"[Other Term]
"Registries"[Mesh]
"Database Management Systems"[Mesh]
"Information Systems"[MeSH Major Topic]
"Registries"[Other Term]
"Information Storage and Retrieval"[MeSH Major Topic]
"Registry"[Title]
"National Program of Cancer Registries"[Mesh]
"Registries"[MeSH Major Topic]
"Information Science"[Mesh]
"Data Management"[Mesh]Final Pubmed Query:("Breast Neoplasms"[Mesh] OR "Breast Cancer"[Title] OR "Breast Neoplasms"[Title] OR "Breast Neoplasms"[Other Term] OR "Breast Cancer"[Other Term]) AND ("Registries"[MeSH Major Topic] OR "Database Management Systems"[MeSH Major Topic] OR "Information Systems"[MeSH Major Topic] OR "Registry"[Other Term] OR "Registry"[Title] OR "Information Storage and Retrieval"[MeSH Major Topic])url:https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=("Breast+Neoplasms"[Mesh]+OR+"Breast+Cancer"[Title]+OR+"Breast+Neoplasms"[Title]+OR+"Breast+Neoplasms"[Other+Term]+OR+"Breast+Cancer"[Other+Term])+AND+("Registries"[MeSH+Major+Topic]+OR+"Database+Management+Systems"[MeSH+Major+Topic]+OR+"Information+Systems"[MeSH+Major+Topic]+OR+"Registry"[Other+Term]+OR+"Registry"[Title]+OR+"Information+Storage+and+Retrieval"[MeSH+Major+Topic])&retmode=json&retstart=1&retmax=10You can download the result of this network and the relationship between the article and the keyword ingraphdictformat fromhere. Manipulated, you can download this graph ingramphmlformat fromhere.EHRIt is not yet complete.Graph VisualizationVarious tools have been developed to visualize graphs. We have done abrief reviewand selected a few tools to use in this program.Graph AnalysisIn this project, we used one of the most powerful libraries for graph analysis. UsingNetworkX, we generated many indicators to check a citation graph. Some materials in this regard are givenhere. You can use other libraries as well.Knowledge ExtractionIn the architecture of this software, the structure of the article is stored in the database and this structure also contains the summary of the article. For this reason, it is possible to perform NLP processes such as keywords extraction, topic extraction etc., which can be completed in the future.Related ArticleThis topic is very interesting from a research point of view, so I have included the articles that were interestinghere.Code QualityWe used flake8 and black libraries to increase code quality.
More information can be foundhere.CitationIf you useTriple Afor your scientific work, consider citing us! We're published inIEEE.@INPROCEEDINGS{10139229,author={Jafarpour, Maryam and Bitaraf, Ehsan and Moeini, Ali and Nahvijou, Azin},booktitle={2023 9th International Conference on Web Research (ICWR)},title={Triple A (AAA): a Tool to Analyze Scientific Literature Metadata with Complex Network Parameters},year={2023},volume={},number={},pages={342-345},doi={10.1109/ICWR57742.2023.10139229}}LicenseTripleA is available under theApache License.