Yes, to fully understand the story, you must start from the beginning of the series and reach its end in order.
If you decide to see the first three X-Men films in chronological sequence, you should get a sense of the wildly uneven quality that eventually characterizes the whole series. It will also protect you from being confused too early in the process.