Removed the builder and made the internal context object mutable.

Also various fix ups from PR review