regexp/syntax: reject large repetitions created by nesting small ones

Fixes issue 7609.

LGTM=r
R=r
CC=golang-codereviews
https://codereview.appspot.com/150270043
CONTRIBUTORS: add Raul Silvera (Google CLA)

Raul wrote the pprof code in CL 153750043.

LGTM=bradfitz, r
R=r, bradfitz
CC=golang-codereviews
https://codereview.appspot.com/146450043