Only do custom ordering on M2M if a custom ordering is set.