(API Change) obs_reset_video: Use return codes

Changed API functions:
libobs: obs_reset_video

Before, video initialization returned a boolean, but "failed" is too
little information, if it fails due to lack of device capabilities or
bad video device parameters, the front-end needs to know that.

The OBS Basic UI has also been updated to reflect this API change.
Check for valid sizes when initializing video
If Direct3D doesn't initialize, try OpenGL