If you're keen to avoid D&D but want to remain rooted in the world of fantasy, we'd recommend starting with Pathfinder or The One Ring. The former is based on an older version of Dungeons & Dragons so retains a lot of its best bits without the baggage, while the latter nails the tone and sense of adventure from its inspiration.
As for sci-fi, Starfinder is an excellent choice. This is made by the same people behind Pathfinder and has been very well-received. Meanwhile, Alien: The Roleplaying Game is an excellent choice if you want something creepier.