proresenc: Remove unneeded parameters from encode_alpha_plane()

Signed-off-by: Diego Biurrun <diego@biurrun.de>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
proresenc: Report buffer overflow

If the allocated size, despite best efforts, is too small, exit
with the appropriate error.

CC: libav-stable@libav.org

Signed-off-by: Diego Biurrun <diego@biurrun.de>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
proresenc: Realloc if buffer is too small

The buffer allocation may be incorrect (e.g. with an alpha plane),
and currently causes the buffer to be set to NULL by init_put_bits,
causing a crash later on.

So, detect that situation, and if detected, reallocate the buffer
and ask for a sample that shows the problem.

CC: libav-stable@libav.org

Signed-off-by: Diego Biurrun <diego@biurrun.de>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
proresenc: Properly account for alpha plane

The packet buffer allocation considers the alpha channel as DCT-coded,
while it is actually run-coded and thus requires a larger buffer.

CC: libav-stable@libav.org

Signed-off-by: Diego Biurrun <diego@biurrun.de>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>