Couple more optimizations

Sorry, I just can't help it.

Added a HashSet for excluded extensions.
Use Interlocked.Exchange when trashing an old object.