After deleting my ~/.berkshelf and tweaking the order of the Berksfile this seemed to not be needed anymore. Probably related to berkshelf lockfile or something...