infer_test_by_dir cleanup

No actual change in functionality here. I was just unhappy with the way
test-ref was singled out before, and realized an obvious fix.
auto merge of #4168 : mttr/servo/mach_inferred_test_cleanup, r=Manishearth

No actual change in functionality here. I was just unhappy with the way test-ref was singled out before, and realized an obvious fix.