I have solved this by a setting "Monitor" which is string and a mode which only toggles between windowed and full screen: https://github.com/FernetMenta/xbmc/commit/8121d342a3921a3700ad19c98d87dbd4de301a63
I have a couple of other commits which enable hot-plugin of monitors which I can't submit as long we have not dropped SDL:

I completely agree which you that the current settings are confusing. Every time I look at it I have the feeling I start from scratch. Your are right, the current implementation follows Windows style but Linux seems to be more flexible and I think we should not limit functionality here.

Currently we load all resolutions into the vector, even those of monitors XBMC does not run on just to filter out appropriate ones in different places of the code. I have added CDisplaySettings::ClearCustomResolutions and reload resolutions for current monitor every time something has changed.

I would be happy to work with you on this.