More cleaner way to handle empty Differences

Relates to #302