add more configuration options for ignoring duplicate pages