Only detect A/B collisions if we aren't moving upwards