KSP-CKAN/CKAN-core — pull request #26 by mgsdk (2014-12-29 13:14:18+00:00)
Title: Support .tar.gz repositories + file identifier using magic numbers

Issue #2 talks about the possibility of using the .tar.gz files from github instead of the zip files. This adds support for that feature using the same library currently used for supporting zip files.

Originally I planned to do the file checking the lazy way by looking at the extension of the downloaded repository file. However, since the temporary files get a non-descriptive extension on my system when downloaded, I implemented a simple file identifier using [magic numbers](http://en.wikipedia.org/wiki/File_format#Magic_number). Currently it supports zip, tar, tar.gz and can guess at ASCII files. This might come in handy in general for error messages, where CKAN thinks it has downloaded a zip file, but it is a clear text file with an error message from the server (I seem to recall this happening at least once with HyperEdit).

As for possible performance enhancements. My unscientific tests, trying to download the repository from github 10 times as a zip and as a tar.gz, yields an performance increase of ~25% (From 3.4 seconds on average with zip to 2.6 seconds on average with tar.gz).