libFLAC: fix more problems with new window functions

This fixes two problems with handling of out-of-bounds arguments
for the window functions, one of which involving an infinite loop

Signed-off-by: Erik de Castro Lopo <erikd@mega-nerd.com>