encoding: fix potential memory leak in encoding detection