empty listener and speaker list before rebuilding