macosx: drop the potential BDMV folder from a bluray path (fixes #10928)

Only the path to the bluray structure itself is supported.
bluray: fix invalid argument crash for error paths in open