Remove group by columns, optimize lookups in Series