Just call XCloseDisplay.

If something else is making OpenGL calls or GLX calls, then that's a bug in XBMC, but you shouldn't work around it by leaking the display.