Inactive widgets are not used much/anymore and can be permanently deleted from the widget list. You can do that easily through Appearance > Widgets or use a plugin if your theme does not allow customization.