vf_deshake: Fix crash with r*=64

Replace literal numbers by named R_MAX and correct their value

Fixes half of Ticket2443

Based on patch by: João Bernardo <jbvsmo@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>