there is a hidden function in scipy for this

@argriffing pointed out on the numpy list that there's already a hidden function in scipy.signal for this getting the fast padding size, so we can remove this code. https://github.com/scipy/scipy/pull/3144