lib: optionally pass in a filesystem type to sanitize for

In some situations (mostly fuse) we can't determine the type of
filesystem we're writing to, so the best we can do is guess based
on filenames it rejects.
encoder: on illegal filename errors, retry with a sanitized name

https://bugzilla.gnome.org/show_bug.cgi?id=720217