Never sweep the metadata.  We don't go through normal handle locking paths, and threads are not prepared to reopen the metadata.  Just skip it.