Package: citation-crawler
citation-crawlerAsynchronous high-concurrency citation crawler, use with caution!异步高并发引文数据爬虫，慎用Only support Semantic Scholar currently.目前支持从Semantic Scholar上爬references和citationsCrawl papers based on BFS(Breath First Search) from the citation network and connect them into an undirected graph. Each edge is a paper, each node is an author.基于广度优先搜索爬引文数据并将其组织为无向图。图的节点是文章，边是引用关系Neo4J output compatible withdblp-crawlerNeo4J形式的输出和dblp-crawler兼容，可以自动识别相同paper不产生重复节点，并可以匹配数据库中已记录的作者，从而为新加入数据库的paper连接作者Installpipinstallcitation-crawlerUsagepython-mcitation_crawler-h
usage:__main__.py[-h][-yYEAR][-lLIMIT]-kKEYWORD[-pPID][-aAID]{networkx,neo4j}...

positionalarguments:{networkx,neo4j}sub-commandhelpnetworkxWriteresultstoajsonfile.neo4jWriteresulttoneo4jdatabase

optionalarguments:-h,--helpshowthishelpmessageandexit-yYEAR,--yearYEAROnlycrawlthepaperafterthespecifiedyear.-lLIMIT,--limitLIMITLimitationofBFSdepth.-kKEYWORD,--keywordKEYWORDSpecifykeywordrules.-pPID,--pidPIDSpecifiedalistofpaperIdtostartcrawling.-aAID,--aidAIDSpecifiedalistofauthorIdtostartcrawling.python-mcitation_crawlernetworkx-h
usage:__main__.pynetworkx[-h]--destDEST

optionalarguments:-h,--helpshowthishelpmessageandexit--destDESTPathtowriteresults.python-mcitation_crawlerneo4j-husage:__main__.pyneo4j[-h][--authAUTH]--uriURI

optionalarguments:-h,--helpshowthishelpmessageandexit--usernameUSERNAMEAuthusernametoneo4jdatabase.--passwordPASSWORDAuthpasswordtoneo4jdatabase.--uriURIURItoneo4jdatabase.Config environment variablesCITATION_CRAWLER_MAX_CACHE_DAYS_AUTHORS:save cache for a paper authors page (to get authors of a published paper) for how many daysdefault:-1(cache forever, since authors of a paper are not likely to change)CITATION_CRAWLER_MAX_CACHE_DAYS_REFERENCES:save cache for a reference page (to get references of a published paper) for how many daysdefault:-1(cache forever, since references of a paper are not likely to change)CITATION_CRAWLER_MAX_CACHE_DAYS_CITATIONSsave cache for a citation page (to get citations of a published paper) for how many daysdefault:7(citations of a paper may change frequently)CITATION_CRAWLER_MAX_CACHE_DAYS_PAPERsave cache for a paper detail page (to get details of a paper) for how many daysdefault:-1(cache forever, since detailed information of a published paper are not likely to change)CITATION_CRAWLER_MAX_CACHE_DAYS_INIT_AUTHORsave cache for an author page (to init papers from specified author by-a) for how many daysdefault:7(author may publish frequently)HTTP_PROXYSet ithttp://your_user:your_password@your_proxy_url:your_proxy_portif you want to use proxyHTTP_TIMEOUTTimeout for each http request, in secondsHTTP_CONCORRENTConcurrent HTTP requestsdefault:8Write to a JSON filee.g. write tosummary.json:python-mcitation_crawler-kvideo-kedge-p27d5dc70280c8628f181a7f8881912025f808256-a1681457networkx--destsummary.jsonJSON format{"nodes":{"<paperId of a paper in Semantic Scholar>":{"paperId":"<paperId of this paper in Semantic Scholar>","dblp_key":"<dblp id of this paper>","title":"<title of this paper>","year":"int <publish year of this paper>","doi":"<doi of this paper>","authors":[{"authorId":"<authorId of this person in Semantic Scholar>","name":"<name of this person>","dblp_name":["<disambiguation name of this person in dblp>","<disambiguation name of this person in dblp>","<disambiguation name of this person in dblp>","......"]},{......},{......},......]},"<paperId of a paper in Semantic Scholar>":{......},"<paperId of a paper in Semantic Scholar>":{......},......},"edges":[["<paperId of a paper in Semantic Scholar>","<paperId of a reference in the above paper>"],[......],[......],......]Write to a Neo4J databasedockerpullneo4j
dockerrun--rm-it-p7474:7474-p7687:7687-v$(pwd)save/neo4j:/data-eNEO4J_AUTH=noneneo4je.g. write toneo4j://localhost:7687:python-mdblp_crawler-kvideo-kedge-p27d5dc70280c8628f181a7f8881912025f808256-a1681457neo4j--urineo4j://localhost:7687TipsWithout index, NEO4J query will be very very slow. So before you start, you should add some index:CREATE INDEX publication_title_hash_index FOR (p:Publication) ON (p.title_hash);
CREATE INDEX publication_dblp_key_index FOR (p:Publication) ON (p.dblp_key);
CREATE INDEX publication_paper_id_index FOR (p:Publication) ON (p.paperId);
CREATE INDEX person_author_id_index FOR (p:Person) ON (p.authorId);
CREATE INDEX person_dblp_pid_index FOR (p:Person) ON (p.dblp_pid);Get initial paper list or author list from a Neo4J databasepython-mdblp_crawler-kvideo-kedge-a"importlib.import_module('citation_crawler.init').papers_in_neo4j('neo4j://localhost:7687')"neo4j--urineo4j://localhost:7687python-mdblp_crawler-kvideo-kedge-p"importlib.import_module('citation_crawler.init').authors_in_neo4j('neo4j://localhost:7687')"neo4j--urineo4j://localhost:7687importlib.import_moduleis flexible, you can import your own variables through this.