Whether you'll find UX design difficult depends on whether you come from a design background or a development background. Though having a formal background in these fields is not essential, learning without a background in these areas can be challenging.