refactor remove_rare and add rank_threshold

also, a stub for using the mallet diagnostics