Yes, page speed is, in fact, a significant ranking factor for mobile searches. The problem with many sites is that their page loads slowly on mobile. According to ThinkWithGoogle, it takes an average of 22 seconds to load a mobile page fully. This isn't good for the user experience and one reason why bounce rates from mobile users are so high. That's why Google has made it more of a significant factor in the somewhat recent page speed update.