checksum: Use glib for io/checksum and misc fixes

- Fixes support for large files.
- Fixes filenames not being passed in the filename encoding.
- Drops openssl dependency.
- Code cleanup.
- Fix 'unknown command' warning.