Do not append dirs without files in it

Otherwise python distutils --record would list a directory when it
shouldn't.