While longer efforts are often never easy or moderate even, Xert bases the Difficulty Rating on the highest Difficulty Score achieved in the activity.  Difficulty Score increases with sustained high intensity efforts, particularly when under high fatigue (lowered MPA).  These are not likely to occur in longer efforts, or if they do, they are early in the ride when you are fresher and able to produce greater power.  As you accumulate longer term fatigue due to the length of a longer ride, it becomes harder to raise your Difficulty Score due to the inability to sustain high intensity efforts very long.  Consider the Difficulty Rating a reflection of the intensity and not the length of the ride.