Removed removals of types from known

Removed what appeared to be a memory optimization but turned out to be awful for concurrency and appeared to hurt performance as well. The known type collection is apparently shared across threads, so removing items from it in a non-thread-safe manner created all sorts of issues for us.