Revert "Attemt at cleaning up nighty packaging after https://github.com/JuliaLang/julia/pull/5095"

This reverts commit c60e4c5299250535988bd86bdc8feb8b9c96ff9f.