Yes, this is bad.  This is a significant problem for a lot of companies. The problem is you are explicitly asking search engines to visit and index them. When they go to the page they get a 404 not found error which wastes a request. This is very common with some script solutions for hreflang, as they take the same URL and clone it for all markets. Sometimes the CMS automatically creates a placeholder URL for every language or country on the site when a page is built for another language which can create multiple invalid URLs. In other cases, language selectors, JavaScript navigation rules or IP detection/redirect protocols add local market URLs even if they don't exist. The solution varies based on your CMS, but start by looking at URL alias, parent-child, or local page spillover rules. In many cases, these have not been set up correctly. You can fix this by using Hreflang Builder and its testing functionality to ping URLs to ensure they are 200 indexable before they are added to the XML sitemaps.