I Scraped more than 1k Top Machine Learning Github Profiles and this is what I Found
By: Khuyen Tran
Tags: Github, Data Science, Data Analysis, Scraping, Data Visualization

I Scraped more than 1k Top Machine Learning Github Profiles and this is what I Found

Gain Insights from Top Machine Learning Profiles on Github

Motivation

When searching the keyword “machine learning” on Github, I found 246,632 machine learning repositories. Since these are top repositories in machine learning, I expect the owners and the contributors of these repositories to be experts or competent in machine learning. Thus, I decided to extract the profiles of these users to gain some interesting insights into their background as well as statistics.

Photo by Markus Winkler on Unsplash

My Method for Scraping

Tools

To scrape, I use three tools:

Beautiful Soup to extract the URLs of all the repositories under the machine learning tag. Beautiful Soup is a Python library that makes it extremely easy to scrape data from websites. If you are not aware of Beautiful Soup, I wrote a tutorial on how to scrape with Beautiful Soup in this article.

PyGithub to extract the information about the users. PyGithub is a Python library to use the Github API v3. With it, you can manage your Github resources (repositories, user profiles, organizations, etc.) from Python scripts.

Requests to extract the information about the repositories and the links to contributors’ profiles.

Methods

I scrape the owners as well as the top 30 contributors of the top 90 repositories that pop up in the search

By removing duplicates as well as removing the profiles that are organizations like udacity, I obtain a list of 1208 users. For each user, I scrape the 20 data points as listed below

new_profile.info()

Specifically, the first 13 data points are obtained from here

The rest of the data points are obtained from the repositories (excluding forked repositories) of a user:

total_stars is the total number of stars of all repositories

is the total number of stars of all repositories max_star is the maximum star among all repositories

is the maximum star among all repositories forks is the total number of forks of all repositories

is the total number of forks of all repositories descriptions are the descriptions from all repository of a user of all repositories

are the descriptions from all repository of a user of all repositories contribution is the number of contribution within last year

Visualize the Data

Bar graphs

After cleaning the data, it comes to the fun part: data visualization. Visualizing the data can give us many insights about the data. I use Plotly for its ease to create interactive plots

The graph is hard to see because of the long tail of users with followers below 100. We can zoom in the left-most portion of the graph to have a better view of the graph.

As we can see, llSourcell (Siraj Raval) gets the majority of the followers (36261). The next user gets about 1/3 followers of what llSourcell has (12682).

We can do further analysis to determine what percentage of followers top 1% of users get

>>> top_n = int(len(top_followers) * 0.01) 12 >>> sum(top_followers.iloc[0: top_n,:].loc[:, 'followers'])/sum(top_followers.followers) 0.41293075864408607

1% of the top users get 41% of the followers!

The same patterns with other data points such as total_stars, max_star, forks . To have a better look at these columns, we will change the y-axis of these features to a logarithmic scale. contribution ’s y-axis is not changed.

These graphs are similar to Zipf’s law, where statistical distribution in certain data sets, such as words in a linguistic corpus, in which the frequencies of certain words are inversely proportional to their ranks.

For example, the most common word in English is “the,” which appears about one-tenth of the time in a typical text even though it is not as important as other words.

We see Zipf’s law a lot in other rankings such as population ranks of cities in various counties, income rankings, ranks of a number of people purchasing a book, etc. Now we see this pattern again in Github data.

Correlation

But what are the relationships between these data points? Is there a strong relationship between them? I use scatter_matrix to get a big picture of the correlation among these data points.

The data points tend to cluster around the left bottom axis because the majority of the users’ data points lie in this range. There is a strong positive relationship between

Maximum number of stars and the total number of stars (0.939)

Number of forks (from others) and the total number of stars (0.929)

The number of forks and the number of followers (0.774)

The number of followers and the total number of stars (0.632)

Languages

What are the favorite languages of the top machine learning users? What is the percentage of Python, Jupyter Notebook, C, R? We could use a bar chart to find out. To have a better look at the most popular languages, we remove languages that below the count of 10

From the bar chart above, we have the ranking of languages among machine learning users:

Python

JavaScript

HTML

Jupyter Notebook

Shell and so on

Hireable

We use Altair to visualize the percentage of users listing themself as hireable

Locations

To get a sense of where the users are in the world, our next task is to visualize the locations of users. We will use the locations obtained from 31% of users who show their location. Start with extracting a list of locations from the df and locating them with geopy.geocoders.Nominatim

Then use Plotly’s scatter_geo to create a map!

Zoom in to see more details of the locations of all the users.

Word Clouds of Descriptions and Bios

Our data also includes the bios of the users as well as all the descriptions of their repositories. We will use these to answer the questions: what are their main focuses and backgrounds.

Generating Word Clouds could give us a big picture of the words and their frequency using in the descriptions and bios. And creating word clouds with Python cannot be easier with WordCloud!

Make word clouds with bio

The keywords look like what we expect to see from machine learning users.

Share your findings

We have been collecting the plots on the list. It is time to create a report and share what you found! Datapane is an ideal tool for this.

Now we have all the plots that we created in this article on a website hosted by Datapane and ready to share!

Conclusion

The data is obtained from the users and contributors of the first 90 best match repositories in the machine learning keyword. Thus, this data does not guarantee to gather all the top machine learning users in Github.

But I hope you could use this article as the guide or inspiration to scrape your own data and visualize it. You will be most likely surprised by what you found. Data science is impactful and interesting when you can use your knowledge to analyze things around you.

Feel free to fork and play with the code for this article in this Github repo.

I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.

Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these: