Beyond Awesome Lists
By: Justin Gosses
Tags: Geoscience, Catalog, Awesome List, Open Source, D3js

Awesome lists are…. awesome. But could they be even more useful? What if instead of just a curated list you also got a view into a wider community?

What is an Awesome list?

An Awesome List is a community curated list of code projects within a specific domain, application, or use case. You can read more in the “Awesome Manifesto” on sindresorhus/awesome.

They are great places to first look for open source code projects that others have found highly use in your problem space, be it markdown editors, JavaScript data visualization, Jupyter notebook widgets, GraphQL, spatial analysis, robotics, or hundreds of other things. Other places people might discover code are word of mouth, google searches, trolling GitHub, seeing the packages used in other code, or having it recommended by a mentor or peer.

Searching for useful code projects is an investment of time. Anything that shortens that time, and enables a developer to reuse code instead of starting from scratch, offers big rewards in terms of time savings.

What Makes Awesome Lists so Successful?

This is mostly guesswork on my part, but I’d say the success of the “Awesome List” form is some combination of the following characteristics:

Easy to start (Start a repository with a single README.md file) Easy to contribute to (Edit a single line in a markdown file) Fills a need (Every developer searches for new code tools) Builds a community (They have a very large number of contributors. People come back to check the list from time to time. People send links to it to others.)

What do Awesome Lists Not do Well?

As a curated list, they provide a signal of potential value. To be included on an Awesome List, someone has to say a project is “Awesome”.

However, there are a variety of other types of information that would useful to know when evaluating what code project to use or what project to contribute to that Awesome Lists don’t provide. They include:

How popular each code project is? Is the rate this new code project is growing in popularity unusual relative to similar code projects in the past? What code projects are used by other code projects within the domain space as a dependency and therefore you might want to learn about? What code projects share dependencies and probably do related things? What code projects share contributors? What organizations own the most code projects in that domain/problem/solution space?

The interesting thing is, the information to get at all these questions exists, it just doesn’t exist in an easy to access to form most of the time.

All of the questions above can be answered with metadata extracted via GitHub’s or Gitlab’s API. Even the dependency information, which was hard to get at several years ago, is now extracted by code platforms are a service and available from their APIs.

Lawerence Livermore National Laboratory’s Software Catalog’s Explore Pages

A great example of harvesting this type of metadata using GitHub’s API and turning it into insightful visualizations is the explore section of Lawerence Livermore National Laboratory’s Software (LLNL) Catalog.

Wanting to understand similar relationships for subsurface geoscience code, I decided to adapt LLNL’s project such that instead of visualization Lawerence Livermore National Laboratory’s code catalog, it was visualizing the code curated in Software Underground’s Awesome-Open-Geoscience awesome list.

The Awesome-Open-Geoscience awesome list was started by myself and several other members of Software Underground, or SWUNG, in October of 2017. As of May 2021, it has 127 watchers, 720 stars, and 51 different contributors. It the most popular repository if you search “geoscience” on GitHub.com.

Adapting llnl.github.io to make open_geosciene_code_projects_viz wasn’t as easy as cloning it and re-running a script as the repository is less a tool and more a product. I had to remove lots of LLNL specific content, edit file paths, and replace bits of html, CSS, and JavaScript throughout the project. There’s still some work to get it to a place to where anyone else with a list of repositories they want to understand could clone the repository, change some lines in a configuration file, re-run a few scripts, and deploy as a GitHub pages page, but it’s getting there. Switching it from LLNL to SWUNG took me several days of work. Switching it for SWUNG to something else took me less than a day.

Ideally, it would great if it only take a couple hours of work to repopulate all the pages and visualizations for another Awesome list.

Images from the Explore Section of the website.

Beyond “curated list” and into “community level view”

Most of these visualizations are interactive on the actual website. They allow for many questions to answered that provide a community level view of the code projects curated on the Awesome list.