MB-7629: Add check for reduction size in view btree encoding

Change-Id: I678a6535cab3a2fcdfb3a82b59e6438f9174d936
Reviewed-on: http://review.couchbase.org/26499
Tested-by: Fulu Li <fulu@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>