Package: dblp-crawler
dblp-crawlerAsynchronous high-concurrency dblp crawler, use with caution!异步高并发dblp爬虫，慎用！Crawl papers from dblp and connect them into an undirected graph. Each edge is a paper, each node is an author.从dblp爬文章并将其组织为无向图。图的边是文章，节点是作者。Installpipinstalldblp-crawlerUsagepython-mdblp_crawler-h
usage:__main__.py[-h][-yYEAR]-kKEYWORD[-pPID][-jJOURNAL]{networkx,neo4j}...

positionalarguments:{networkx,neo4j}sub-commandhelpnetworkxnetworkxhelpneo4jneo4jhelpoptionalarguments:-h,--helpshowthishelpmessageandexit-yYEAR,--yearYEAROnlycrawlthepaperafterthespecifiedyear.-kKEYWORD,--keywordKEYWORDSpecifykeywordrules.-pPID,--pidPIDSpecifiedauthorpidstostartcrawling.-jJOURNAL,--journalJOURNALSpecifyauthorjournalkeystostartcrawling.python-mdblp_crawlernetworkx-h
usage:__main__.pynetworkx[-h]--destDEST

optionalarguments:-h,--helpshowthishelpmessageandexit--destDESTPathtowriteresults.python-mdblp_crawlerneo4j-husage:__main__.pyneo4j[-h][--authAUTH]--uriURI

optionalarguments:-h,--helpshowthishelpmessageandexit--usernameUSERNAMEAuthusernametoneo4jdatabase.--passwordPASSWORDAuthpasswordtoneo4jdatabase.--uriURIURItoneo4jdatabase.--selectMarkkeyword-matchedpublicationsindatabase(setselected=true).Config environment variablesDBLP_CRAWLER_MAX_CACHE_DAYS_PERSON:save cache for a person page for how many daysdefault:30DBLP_CRAWLER_MAX_CACHE_DAYS_JOURNAL:save cache for a journal page (e.g.IEEE Transactions on Multimedia Volume 25, 2023) or conference page (e.g.31st ACM Multimedia 2023) for how many daysdefault:-1(cache forever)DBLP_CRAWLER_MAX_CACHE_DAYS_JOURNAL_LISTsave cache for a journal list page (e.g.IEEE Transactions on Multimedia) or conference list page (e.g.ACM Multimedia) for how many daysdefault:30HTTP_PROXYSet ithttp://your_user:your_password@your_proxy_url:your_proxy_portif you want to use proxyHTTP_TIMEOUTTimeout for each http request, in secondsHTTP_CONCORRENTConcurrent HTTP requestsdefault:8Write to a JSON filee.g. write tosummary.json:python-mdblp_crawler-kvideo-kedge-pl/JiangchuanLiunetworkx--destsummary.jsonJSON format{"nodes":{// each node is a person"<dblp id of a person>":{"id":"<dblp id of this person>","label":"<name in dblp>","publications":[// selected papers of this person (selected by "-k" and "-y" args)"<dblp id of a paper>","<dblp id of a paper>","<dblp id of a paper>","......"],"person":{// detailed data of this person"dblp_pid":"<dblp id of this person>","name":"<name in dblp>","affiliations":["<affiliation of this person>","<affiliation of this person>","......"],"publications":[// all papers of this person"<dblp id of a paper>","<dblp id of a paper>","<dblp id of a paper>","......"]}},"<dblp id of a person>":{......},"<dblp id of a person>":{......},"<dblp id of a person>":{......},......},"edges":{// each node is a cooperation of two person"<id of this edge>":{"from":"<dblp id of this person 1>","to":"<dblp id of this person 2>","publications":[// selected papers that contain both this two persons as authors (selected by "-k" and "-y" args)"<dblp id of a paper>","<dblp id of a paper>","<dblp id of a paper>","......"],"cooperation":[// all papers that contain both this two persons as authors (selected by "-k" and "-y" args)"<dblp id of a paper>","<dblp id of a paper>","<dblp id of a paper>","......"]},"publications":{// related publications"<dblp id of a paper>":{"key":"<dblp id of this paper>","title":"<title of this paper>","journal":"<name of the journal that this paper published on>","journal_key":"<dblp id of the journal that this paper published on>","year":"int <publish year of this paper>","doi":"<doi of this paper>","ccf":"A|B|C|N <CCF rank of this paper>","authors":{"<dblp id of a person>":{"name":"<name in dblp>","orcid":"<orcid of this person>"},"<dblp id of a person>":{......},"<dblp id of a person>":{......},......},"selected":"true|false <whether the publication is selected (selected by -k and -y args)>"}}}}Write to a Neo4J databasedockerpullneo4j
dockerrun--rm-it--nameneo4j-p7474:7474-p7687:7687-v"$(pwd)/save/neo4j:/data"-eNEO4J_AUTH=noneneo4je.g. write toneo4j://localhost:7687:python-mdblp_crawler-kvideo-kedge-pl/JiangchuanLiuneo4j--urineo4j://localhost:7687Only crawl the paper after specified yeare.g. crawl the paper after 2016 (include 2016)python-mdblp_crawler-kvideo-kedge-pl/JiangchuanLiu-y2016networkx--destsummary.jsonKeywords with two or more wordse.g. super resolution (publications with title contains both "super" and "resolution" will be selected)python-mdblp_crawler-kvideo-kedge-pl/JiangchuanLiu-k"'super','resolution'"networkx--destsummary.jsonInit authors from journale.g. init authors from ACM MM and MMSys (db/conf/mmis the key for ACM MM in dblp: "https://dblp.org/db/conf/mm/index.xml",db/conf/mmsysis the key for MMSys in dblp: "https://dblp.org/db/conf/mmsys/index.xml")python-mdblp_crawler-kvideo-kedge-jdb/conf/mm-jdb/conf/mmsysnetworkx--destsummary.jsonInit authors from journal in some variablese.g. there is aCCF_Aindblp_crawler.datacontains keys of CCF A conferences, and MMSys is also great but not in CCF Apython-mdblp_crawler-kvideo-kedge-j"importlib.import_module('dblp_crawler.data').CCF_A"-jdb/conf/mmsysnetworkx--destsummary.jsonimportlib.import_moduleis flexible, you can import your own variables through this.Init authors in some variablese.g. want to crawl publications of those authors stored in neo4j databasepython-mdblp_crawler-kvideo-kedge-p"importlib.import_module('dblp_crawler.data').authors_in_neo4j('neo4j://localhost:7687')"networkx--destsummary.jsonimportlib.import_moduleis flexible, you can import your own variables through this.dblp_crawler.filterUsageCrawling papers takes a long time, so do not filter the papers in the crawling process. Instead, use a separate programdblp_crawler.filter` to filter the papers.python-mdblp_crawler.filter-husage:__main__.py[-h]-iINPUT-oOUTPUT-fFILTER

optionalarguments:-h,--helpshowthishelpmessageandexit-iINPUT,--inputINPUTInputfilepath.-oOUTPUT,--outputOUTPUTOutputfilepath.-fFILTER,--filterFILTERFilterfunctions.Use internal filtere.g.drop_old_publicationsis an internal function that drop publication by yearpython-mdblp_crawler.filter-isummary.json-osummary.filter.json-f"lambda summary: drop_old_publications(summary, 2016)"Use multiple filterse.g.drop_old_publicationsis an internal function that drop publications by year;drop_nodes_by_all_publicationsis an internal function that drop nodes by the sum of publicationspython-mdblp_crawler.filter-isummary.json-osummary.filter.json\-f"lambda summary: drop_old_person_publications(summary, 2018)"\-f"lambda summary: drop_old_cooperation(summary, 2018)"\-f"lambda summary: drop_nodes_by_all_publications(summary, 4)"\-f"lambda summary: drop_edges_by_all_publications(summary, 4)"Use your own filtere.g. another method to use-f "lambda summary: drop_old_publications(summary, 2016)"python-mdblp_crawler.filter-isummary.json-osummary.filter.json-f"lambda summary: importlib.import_module('dblp_crawler.filter').drop_old_publications(summary, 2016)"importlib.import_moduleis flexible, you can import your own variables through this.