Revert "Move "lazy initialization" code from XlibBackend to TouchpadBackend"

This reverts commit a0311d15a221d6dd6e28bd50047af367c879ed21.
Too lazy to split this into smaller commits