Bug fix for allowing the terms panel to ignore it's on fitlers.

This fixes an issue in the "Allow the Terms panel to ignore it's own filters" #1201 pull request.

The terms panel would lose track of it's filters if multiple term panels and filters were added and removed.