infer_test_by_dir cleanup

No actual change in functionality here. I was just unhappy with the way test-ref was singled out before, and realized an obvious fix.