Package: top-github-scraper
Top Github ScraperScrape top Github repositories and users based on keywords.I used this tool to analyze the top 1k machine learning users inthis article.SetupInstallationpipinstalltop-github-scraperAdd CredentialsTo make sure you can scrape many repositories and users, add your GitHub's credentials to.envfile.touch.envAdd your username andtokento.envfile:GITHUB_USERNAME=yourusernameGITHUB_TOKEN=yourtokenUsageGet Top Github Repositories' URLsfromtop_github_scraperimportget_top_repo_urlsget_top_repo_urls(keyword="machine learning",stop_page=10)Output attop_repo_urls_<keyword>_<sort_by>_<start_page>_<end_page>.json:["/josephmisiti/awesome-machine-learning","/wepe/MachineLearning","/udacity/machine-learning","/Jack-Cherish/Machine-Learning","/ZuzooVn/machine-learning-for-software-engineers","/rasbt/python-machine-learning-book","/lawlite19/MachineLearning_Python","/lazyprogrammer/machine_learning_examples","/trekhleb/homemade-machine-learning","/ujjwalkarn/Machine-Learning-Tutorials"]Get Top Github Repositories' Informationfromtop_github_scraperimportget_top_reposget_top_repos("machine learning",stop_page=10)Output for 1 repository attop_repo_info_<keyword>_<sort_by>_<start_page>_<end_page>.json:{"stargazers_count":48620,"forks_count":12155,"contributors":{"login":["josephmisiti","josephmmisiti","hslatman","0asa","ajkl","ipcenas","cogmission","spekulatius","basickarl","NathanEpstein"],"url":["https://api.github.com/users/josephmisiti","https://api.github.com/users/josephmmisiti","https://api.github.com/users/hslatman","https://api.github.com/users/0asa","https://api.github.com/users/ajkl","https://api.github.com/users/ipcenas","https://api.github.com/users/cogmission","https://api.github.com/users/spekulatius","https://api.github.com/users/basickarl","https://api.github.com/users/NathanEpstein"],"contributions":[671,105,21,12,11,9,8,7,7,7]}}Get Top Github Contributors' Profilesfromtop_github_scraperimportget_top_contributorsget_top_contributors("machine learning",stop_page=10)Output attop_contributor_info_<keyword>_<sort_by>_<start_page>_<end_page>.csv:loginurltypenamecompanylocationemailhireablebiopublic_repospublic_gistsfollowersfollowing0josephmisitihttps://api.github.com/users/josephmisitiUserJoseph MisitiMath & Pencil"Brooklyn, NY"TrueMathematician & Co-founder of Math & Pencil22914227052751josephmmisitihttps://api.github.com/users/josephmmisitiUser00202hslatmanhttps://api.github.com/users/hslatmanUserHerman SlatmanDistributIT133204696730asahttps://api.github.com/users/0asaUserVincent BottaBelgium"Innovation Engineer @evs-broadcast, previously Data Scientist @kensuio, E-Marketing Tools Manager @Diagenode, cofounder @Antibody-Adviser and photographer"351525164ajklhttps://api.github.com/users/ajklUserAjinkya Kalekaleajinkya@gmail.com5812945ipcenashttps://api.github.com/users/ipcenasUser790106cogmissionhttps://api.github.com/users/cogmissionUserDavid RayThird planet from the sun...cognitionmission@gmail.comHumanity's freedom and abundance through the pursuit of technological innovation in the area of cognitive applications - Cognition Mission301954447spekulatiushttps://api.github.com/users/spekulatiusUserPeter Thaleikis@bringyourownideas127.0.0.1TrueSoftware engineer focused on solutions using open source and simply filling in the gaps to fulfill the requirements.4212329208basickarlhttps://api.github.com/users/basickarlUserKarl Morrison"Malmö, Sweden"karl@basickarl.ioThe question is: Will you take me seriously511269NathanEpsteinhttps://api.github.com/users/NathanEpsteinUserNathan Epstein"New York, NY"nathanepst@gmail.comTrue23122080Get Top Github Users' Profilesfromtop_github_scraperimportget_top_usersget_top_users("machine learning",stop_page=10)Output attop_user_info_<keyword>_<start_page>_<end_page>.csvloginurltypenamecompanylocationemailhireablebiopublic_repospublic_gistsfollowersfollowing0rasbthttps://api.github.com/users/rasbtUserSebastian RaschkaUW-Madison"Madison, WI""Machine Learning researcher & open source contributor. Author of ""Python Machine Learning."" Asst. Prof. of Statistics @ UW-Madison."71513888351tqchenhttps://api.github.com/users/tqchenUserTianqi Chen"CMU, OctoML"Large scale Machine Learning28186111262halfrosthttps://api.github.com/users/halfrostUserhalfrost@AlibabaShanghai Chinai@halfrost.com💪天道酬勤，勤能补拙。博观而约取，厚积而薄发。Gopher / Rustacean / iOS Dev. / Machine Learning / Retired acmer / Math / Philosophy / Technical Writer.22085663143ageronhttps://api.github.com/users/ageronUserAurélien GeronParisAuthor of the book Hands-On Machine Learning with Scikit-Learn and TensorFlow. Former PM of YouTube video classification and founder & CTO of a telco operator.4316838324chiphuyenhttps://api.github.com/users/chiphuyenUserChip Huyenhttps://snorkel.ai"Mountain View, CA"TrueDeveloping tools and best practices for machine learning production.1917839155rhieverhttps://api.github.com/users/rhieverUserRandy OlsonFOXO BioScience"Vancouver, WA"rso@randalolson.com"Chief Data Scientist, @FOXOBioScience. AI, Machine Learning, and Data Visualization specialist. Community leader for /r/DataIsBeautiful."77175363136lexfridmanhttps://api.github.com/users/lexfridmanUserLex FridmanMIT"Cambridge, MA""AI researcher working on autonomous vehicles, human-robot interaction, and machine learning at MIT and beyond."20503107eriklindernorenhttps://api.github.com/users/eriklindernorenUserErik Linder-Norén"Stockholm, Sweden"eriklindernoren@gmail.com"ML engineer at Apple. Excited about machine learning, basketball and building things."2403764118roboticcamhttps://api.github.com/users/roboticcamUserA/Prof Richard Xu                 徐亦达教授University of Technology SydneySydney Australia"I am an A/Professor in Machine Learning at UTS. manage a large research team of postdoc, PhD students close to 30 people"100356109ogriselhttps://api.github.com/users/ogriselUserOlivier GriselInria"Paris, France"olivier.grisel@ensta.orgMachine Learning Engineer a Inria Saclay (Parietal team).174933237116ParametersView a full list of paramtershere.How the Data is Scrapedtop-github-scraperscrapes the owners as well as the contributors of the top repositories that pop up in the search when searching for a specific keyword on GitHub.For each user,top-github-scraperscrapes 16 data points:login: usernameurl: URL of the usertype: Whether this account is a user or an organizationname: Name of the usercompany: User's companylocation: User's locationemail: User's emailhireable: Whether the user is hireablebio: Short description of the userpublic_repos: Number of public repositories the user has (including forked repositories)public_gists: Number of public repositories the user has (including forked gists)followers: Number of followers the user hasfollowing: Number of people the user is following