What I was seeing was that in development mode, the menu items would get completely reloaded and all of what was added by the plugin on initial boot of Foreman would be lost from that point on.