This `return` in the middle of the method makes this not memoize correctly.  My suggestion is to split this into two methods:

``` ruby
  def slowest_groups
    @slowest_groups ||= calculate_slowest_groups
  end

private

  def calculate_slowest_groups
    # the original `slowest_groups` code
  end
```

`calculate_slowest_groups` can have early returns and it doesn't cause any problems for the memoization.