ENH: Slowly getting some more speedups