@popcornmix: Yeah it's maxing out on cpu and the busy indicator seems to spin forever so there's still some activity but the video never starts playing. I've obviously tested all cases with the same file(s). The problem is debugging this on RPi itself.

Do note that I also tested sftp + always-buffer and that does work (although video starting is quite slow). My best guess is that due to the cpu maxing out, one (or more) threads (either DVDPlayer's or Curl's) die off.