Fix some double-frees/use-after-free

r? @frank-trampe
Does pfb get used later somehow?