TST: make pytables tests go thru a temporary dir and file

TST/BUG: incorrect way of testing for r+ modes

related #5419