Remove need to always build go math package