Remove handling for [], {} in nested option

The "allButNested" option allowed [] and {} to be nested inside ().
This functionality differed from the "allButNested" variant of the
[] and {} rules. The current functionality follows the functionality
of the previous rules and only allows the same character (parentheses)
to be nested, ie: (( 1 + 2 ) * ( 3 + 4 ))