Thanks for the feedback, I was originally thinking of the plugins/modules kind of like traits.
Allowing the user to modify active plugins/modules does allow for more flexibility, especially for admin users.

This is related the @FranciscoCanas  comment on decoupling observers and the package view.
Observer and package view will be decoupled to accomodate this use case.

We can verify users permissions to add/remove an observer from a package from the observers page if needed.
I think we should go with this idea.