Fix #1304 Aggregates with categorical/xref filter broken