Clear out the log handlers when initializing the factory