ENH: don't cache fast operations

Address @AlexandreAbraham's comment