Q:

Two options for changing modes.  Which one is better? 

The system, that i'm working on, has two different content density styles: compact and cozy.  The first mode is for running on non-touch (mouse and keyboard-enabled) devices, with reduced control dimensions.  The second one is for touch devices, where all control dimensions are big enough for finger tips. 
The problem is that this system is working with apps, and not all of them support these modes.  Thereby, the embedded controls in a such app can be changed when switching between modes, but custom controls cannot. 
There are two possible solutions:

Enforce mode only if an app supports the required mode.  But then we'll get a 'jumping' behavior, when navigating from a home screen in a cozy mode to an app in a compact mode. 
Always enforce mode even if an app does not support the required mode.  But then the apps that don't support the mode will look 'broken'. 

Which on is better?  Or are there any other solutions, that I don't see? 

A:

I think it's really important not to 'penalise' the user based on their screen that they are using.   So, even if the application doesn't work at 100% on certain devices, it's important to give the user the choice to at least use it, rather than get frustrated and have to go look for another device. 
I think something that has a jumping behaviour sounds more like a cosmetic issue - in that it mightn't look 100%, but should retain most of it's functionality, is better than a mode where the app is 'broken'. 
I hope I'm explaining your issue correctly - I've seen this behaviour in some mobile optimised sites, where the mobile site only features links to certain 'mobile' pages, and the pages which don't have a mobile optimised version link back to the desktop site.   Hard to use on mobile, but at least I can use it! 
Also worth nothing, these sites typically use an icon such as:

to show that it's linking off the mobile site.