PERF: Fixed decoding perf issue on pyt3 (GH5441)