Yes, absolutely. There are rare fish that only show up in specific locations, rare fish that only pop up in certain seasons, and those that only come out when it is raining. They aren't legendary in terms of difficulty but you will have to know where and when to find them. Here is a quick list of some examples of the rarest fish in Stardew Valley • Spook Fish • Lava Eel • Woodskip • Sandfish • Ghost Fish