Stop open-files from dramatically slowing down tests 

After #3096 was merged, the travis builds have slowed down rather dramatically (from net ~2 hr to ~3 hr).  This seems to be due to something that the open file checking is doing on some systems and not others.

There was some discussion after the merge for #3096, but it seems like this is probably a bug somewhere, so I'm opening this as a separate issue for the tracker.  @sargas reported that he can reproduce it locally, but isn't sure how to profile pytest and determine exactly where in the code it's slowing down.  So any hints from anyone on this would be appreciated...

cc @sargas @astrofrog @embray