you can toggle a rift group off by looking at tops of towers

but you must look at the buttons on the group to turn them on