Calo towers speed-up

Forward port of #1885 which itself is a rebase of #1404.