tls: Fix encoding handling in certificate-related functions (GH-8366)