Only setup the master specs repo if required

Should hopefully fix #2562 and CocoaPods/Core#173