Inactive themes should be OK with the speed of your WordPress website. When you switch to a different theme, the inactive theme is not being used on your website, so it does not interact with the core code or make any database queries. However, keeping your themes up to date is still a good idea to ensure they are secure.