Optimize everything, stop releasing memory

Reading xattrs is so damn slow that we need to speed up everything else
to compensate.
Preallocate buffer to minimize getxattr() calls
Gobble memory like there's no tomorrow