Cleanup this logic a bit more

I'm not convinced that nil is a value for a key we would need to convert
so this should be a perfectly fine refactoring.