Are Those Who Does Not Pray Will Be Regarded As "polytheists"?
The verse you quoted starts with its context from here: So, [now that these facts have become evident,] following one God continue to point your face [like your father Abraham] towards his religion. Follow the nature created by God [O Prophet!] on which He has created people. This nature created by God cannot be changed. This only is the straight path but most people know not. [Follow it] by turning towards God all of you and keep fearing Him and be diligent in the prayer and be not among the idolaters, who created divisions in their religion and became divided into factions. Each faction is engrossed in what it has. So the establishment of prayer to God only is the demand of monotheistic behavior. Since the first and immediate addressees of the Quran and the prophet were polytheists. Not being monotheistic and not ready to offer prayer to God only meant that the person is a polytheist. The same describes the Hadith. This attitude is kufr (denial) too because they were snubbing the message of God to worship Him and worship Him alone. Today, the if a Muslim knowingly leaves praying despite his knowledge that how important it is, it is the same attitude of kufr if not a polytheistic attitude: giving no importance to what God wants him to give importance.