As long as an ereader has Bluetooth connectivity, you should be able to pair a set of wireless headphones to listen to supported audio files. While brands like Onyx and PocketBook will allow you to sideload audio files, including audiobooks, Kindle and Kobo devices only support those purchased from their respective stores.
In fact, some Onyx and PocketBook ereaders have built-in speakers, so you may not even need headphones to listen. However, don't expect great sound quality from them because, like your phone, they're small drivers and the sound would be better via a connected pair of headphones.
Also keep in mind that audio files are typically larger than ebook formats, so if you're storing a lot of these on your device, you'll need to keep an eye on how much space you've got left.