[encryption] improved error handling

Add some encryption specific exceptions to handle various error cases, including #9636