Only remove markers when directions are possible