ArchiveTeam/github-download-grab — pull request #1 by soult (2012-12-14 00:58:03+00:00)
Title: Decode htmlentities in download URLs

As balrog_ discovered, wget does not always download all URLs. It turns out that file URLs can contain characters such as &, which will be htmlencoded as &. The lua script extracts the download URLs from HTML but does not decode HTML entities, which causes the URLs to contain the encoded entities. Wget tries to download the file, receives a 404 and gives up, resulting in incomplete downloads.

My patch decodes all HTML entities decoded by Ruby's CGI.escapeHTML (which Github probably uses).

A simple script that compares the index.txt file with the downloaded files should be able to output a list of users for which the downloads were not complete.