Fix NVENC reference counting if encoder initialization fails