proresenc: Remove unneeded parameters from encode_alpha_plane()

Signed-off-by: Diego Biurrun <diego@biurrun.de>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit b16699f2da9c1d41eff852ec3a0c81f74fd44421)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
proresenc: Report buffer overflow

If the allocated size, despite best efforts, is too small, exit
with the appropriate error.

CC: libav-stable@libav.org

Signed-off-by: Diego Biurrun <diego@biurrun.de>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 58b68e4fdea22e22178e237bda950b09cc6f363a)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
proresenc: Realloc if buffer is too small

The buffer allocation may be incorrect (e.g. with an alpha plane),
and currently causes the buffer to be set to NULL by init_put_bits,
causing a crash later on.

So, detect that situation, and if detected, reallocate the buffer
and ask for a sample that shows the problem.

CC: libav-stable@libav.org

Signed-off-by: Diego Biurrun <diego@biurrun.de>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 45ce880a9b3e50cfa088f111dffaf8685bd7bc6b)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
proresenc: Properly account for alpha plane

The packet buffer allocation considers the alpha channel as DCT-coded,
while it is actually run-coded and thus requires a larger buffer.

CC: libav-stable@libav.org

Signed-off-by: Diego Biurrun <diego@biurrun.de>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 41e1354c101004ccd46dc08d3dd6e956e83a6b51)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>