nut: Fix unchecked allocations

CC: libav-stable@libav.org

(cherry picked from commit b1fcdc08ceb5df69fac34aa0d57c56905d32b8b4)
Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
prores: Add a codepath for decoding errors

(cherry picked from commit 44690dfa683f620c77e9f0e8e9bc5682608636b1)
Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>