Use xattrs again instead of filename to store size

I ended up having way too many O(n) readdir() operations any time we
wanted to look at a file.